2022-08-17T12:06:33.5827038Z Requested labels: linux.2xlarge 2022-08-17T12:06:33.5827125Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/82657/merge 2022-08-17T12:06:33.5827153Z Waiting for a runner to pick up this job... 2022-08-17T12:06:36.1542426Z Job is about to start running on the runner: i-068af22e3918a20a1 (repository) 2022-08-17T12:06:42.2422142Z Current runner version: '2.295.0' 2022-08-17T12:06:42.2427618Z Runner name: 'i-068af22e3918a20a1' 2022-08-17T12:06:42.2428067Z Runner group name: 'Default' 2022-08-17T12:06:42.2428642Z Machine name: 'ip-10-0-2-81' 2022-08-17T12:06:42.2430447Z ##[group]GITHUB_TOKEN Permissions 2022-08-17T12:06:42.2431047Z Actions: read 2022-08-17T12:06:42.2431316Z Checks: read 2022-08-17T12:06:42.2431574Z Contents: read 2022-08-17T12:06:42.2431825Z Deployments: read 2022-08-17T12:06:42.2432095Z Discussions: read 2022-08-17T12:06:42.2432358Z Issues: read 2022-08-17T12:06:42.2432597Z Metadata: read 2022-08-17T12:06:42.2432855Z Packages: read 2022-08-17T12:06:42.2433109Z Pages: read 2022-08-17T12:06:42.2433374Z PullRequests: read 2022-08-17T12:06:42.2433783Z RepositoryProjects: read 2022-08-17T12:06:42.2434087Z SecurityEvents: read 2022-08-17T12:06:42.2434345Z Statuses: read 2022-08-17T12:06:42.2434607Z ##[endgroup] 2022-08-17T12:06:42.2437661Z Secret source: None 2022-08-17T12:06:42.2438169Z Prepare workflow directory 2022-08-17T12:06:42.3379647Z Prepare all required actions 2022-08-17T12:06:42.3541140Z Getting action download info 2022-08-17T12:06:42.5680748Z Download action repository 'pytorch/pytorch@master' (SHA:2a096e940d33a33c4eb6df1c2ed4da607bd31a7f) 2022-08-17T12:06:45.1901416Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a) 2022-08-17T12:06:45.3712949Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886) 2022-08-17T12:06:45.6785260Z Getting action download info 2022-08-17T12:06:45.8420059Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246) 2022-08-17T12:06:46.1748544Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-08-17T12:06:46.1748853Z with: 2022-08-17T12:06:46.1749048Z submodules: recursive 2022-08-17T12:06:46.1749243Z fetch-depth: 0 2022-08-17T12:06:46.1749420Z env: 2022-08-17T12:06:46.1749617Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:06:46.1749803Z ##[endgroup] 2022-08-17T12:06:46.1976698Z ##[group]Run retry () { 2022-08-17T12:06:46.1976966Z retry () { 2022-08-17T12:06:46.1977215Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-08-17T12:06:46.1977445Z } 2022-08-17T12:06:46.1977636Z echo "${GITHUB_WORKSPACE}" 2022-08-17T12:06:46.1977872Z if [ -z "${NO_SUDO}" ]; then 2022-08-17T12:06:46.1978115Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-08-17T12:06:46.1978335Z else 2022-08-17T12:06:46.1978534Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-08-17T12:06:46.1978737Z fi 2022-08-17T12:06:46.1978976Z mkdir "${GITHUB_WORKSPACE}" 2022-08-17T12:06:46.1994782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:06:46.1995038Z env: 2022-08-17T12:06:46.1995235Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:06:46.1995418Z NO_SUDO: 2022-08-17T12:06:46.1995607Z ##[endgroup] 2022-08-17T12:06:46.2179459Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-08-17T12:06:46.2487245Z ##[group]Run malfet/checkout@silent-checkout 2022-08-17T12:06:46.2487486Z with: 2022-08-17T12:06:46.2487750Z ref: ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T12:06:46.2487978Z fetch-depth: 0 2022-08-17T12:06:46.2488159Z submodules: recursive 2022-08-17T12:06:46.2488357Z quiet-checkout: true 2022-08-17T12:06:46.2488566Z repository: pytorch/pytorch 2022-08-17T12:06:46.2488958Z token: *** 2022-08-17T12:06:46.2489130Z ssh-strict: true 2022-08-17T12:06:46.2489333Z persist-credentials: true 2022-08-17T12:06:46.2489533Z clean: true 2022-08-17T12:06:46.2489704Z lfs: false 2022-08-17T12:06:46.2489895Z set-safe-directory: true 2022-08-17T12:06:46.2490176Z env: 2022-08-17T12:06:46.2490386Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:06:46.2490617Z ##[endgroup] 2022-08-17T12:06:46.3650877Z Syncing repository: pytorch/pytorch 2022-08-17T12:06:46.3652453Z ##[group]Getting Git version info 2022-08-17T12:06:46.3653246Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-08-17T12:06:46.3653767Z [command]/usr/bin/git version 2022-08-17T12:06:46.3654095Z git version 2.37.1 2022-08-17T12:06:46.3654828Z ##[endgroup] 2022-08-17T12:06:46.3667389Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/67bdd8f2-b040-47d3-bd56-2801b4567ed6' before making global git config changes 2022-08-17T12:06:46.3667913Z Adding repository directory to the temporary git global config as a safe directory 2022-08-17T12:06:46.3668470Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-08-17T12:06:46.3702276Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-08-17T12:06:46.3705888Z ##[group]Initializing the repository 2022-08-17T12:06:46.3743686Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-08-17T12:06:46.3744469Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-08-17T12:06:46.3745102Z hint: is subject to change. To configure the initial branch name to use in all 2022-08-17T12:06:46.3745519Z hint: of your new repositories, which will suppress this warning, call: 2022-08-17T12:06:46.3745960Z hint: 2022-08-17T12:06:46.3746518Z hint: git config --global init.defaultBranch 2022-08-17T12:06:46.3747140Z hint: 2022-08-17T12:06:46.3747455Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-08-17T12:06:46.3747837Z hint: 'development'. The just-created branch can be renamed via this command: 2022-08-17T12:06:46.3748065Z hint: 2022-08-17T12:06:46.3748418Z hint: git branch -m 2022-08-17T12:06:46.3748819Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2022-08-17T12:06:46.3749434Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-08-17T12:06:46.3778834Z ##[endgroup] 2022-08-17T12:06:46.3779246Z ##[group]Disabling automatic garbage collection 2022-08-17T12:06:46.3783571Z [command]/usr/bin/git config --local gc.auto 0 2022-08-17T12:06:46.3808657Z ##[endgroup] 2022-08-17T12:06:46.3809020Z ##[group]Setting up auth 2022-08-17T12:06:46.3816292Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-08-17T12:06:46.3844132Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-08-17T12:06:46.4103634Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-08-17T12:06:46.4134585Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-08-17T12:06:46.4385314Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-08-17T12:06:46.4424083Z ##[endgroup] 2022-08-17T12:06:46.4424492Z ##[group]Fetching the repository 2022-08-17T12:06:46.4430822Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-08-17T12:07:32.0802364Z [command]/usr/bin/git rev-parse --verify --quiet ce6a3c605df99d1df57c0dda75c06d748e54ed2a^{object} 2022-08-17T12:07:32.0834261Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T12:07:33.1128848Z ##[endgroup] 2022-08-17T12:07:33.1129427Z ##[group]Determining the checkout info 2022-08-17T12:07:33.1130281Z ##[endgroup] 2022-08-17T12:07:33.1130672Z ##[group]Checking out the ref 2022-08-17T12:07:33.1136047Z [command]/usr/bin/git checkout --quiet --force ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T12:07:34.3668657Z ##[endgroup] 2022-08-17T12:07:34.3669289Z ##[group]Setting up auth for fetching submodules 2022-08-17T12:07:34.3674756Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-08-17T12:07:34.3717070Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-08-17T12:07:34.3744511Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-08-17T12:07:34.3770976Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-08-17T12:07:34.3795067Z ##[endgroup] 2022-08-17T12:07:34.3795440Z ##[group]Fetching submodules 2022-08-17T12:07:34.3798637Z [command]/usr/bin/git submodule sync --recursive 2022-08-17T12:07:34.4056103Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-08-17T12:07:34.4305347Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-08-17T12:07:34.4306227Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-08-17T12:07:34.4308000Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-08-17T12:07:34.4310544Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-08-17T12:07:34.4313075Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-08-17T12:07:34.4315901Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-08-17T12:07:34.4318707Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-08-17T12:07:34.4321426Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-08-17T12:07:34.4324377Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-08-17T12:07:34.4328416Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-08-17T12:07:34.4331295Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-08-17T12:07:34.4334521Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-08-17T12:07:34.4337943Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-08-17T12:07:34.4341165Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-08-17T12:07:34.4344704Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-08-17T12:07:34.4348477Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:07:34.4352210Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-08-17T12:07:34.4356123Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-08-17T12:07:34.4360006Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-08-17T12:07:34.4364017Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-08-17T12:07:34.4368619Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-08-17T12:07:34.4372950Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-08-17T12:07:34.4377345Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-08-17T12:07:34.4381833Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-08-17T12:07:34.4386354Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-08-17T12:07:34.4391041Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-08-17T12:07:34.4395981Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-08-17T12:07:34.4400804Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-08-17T12:07:34.4405689Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-08-17T12:07:34.4410674Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-08-17T12:07:34.4415725Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-08-17T12:07:34.4420939Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-08-17T12:07:34.4426275Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-08-17T12:07:34.4431702Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-08-17T12:07:34.4437504Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-08-17T12:07:34.4442914Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-08-17T12:07:34.4448869Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-08-17T12:07:34.4454691Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-08-17T12:07:34.4460639Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-08-17T12:07:34.4489448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-08-17T12:07:34.7507455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-08-17T12:07:34.9457609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-08-17T12:07:35.1342826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-08-17T12:07:35.4509400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-08-17T12:07:35.7015516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-08-17T12:07:41.1394187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-08-17T12:07:41.5007440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-08-17T12:07:42.0061920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-08-17T12:07:43.5304125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-08-17T12:07:44.6165752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-08-17T12:07:51.2455517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-08-17T12:07:51.8773820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-08-17T12:07:53.2095413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-08-17T12:07:54.4418733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-08-17T12:07:54.6456874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-08-17T12:07:55.1346180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-08-17T12:07:55.4343537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-08-17T12:07:56.4616811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-08-17T12:07:56.8428739Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-08-17T12:07:57.1395295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-08-17T12:07:57.3834507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-08-17T12:07:58.7851898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-08-17T12:07:59.2063376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-08-17T12:07:59.5524926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-08-17T12:08:05.7197811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-08-17T12:08:07.6454412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-08-17T12:08:08.0501266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-08-17T12:08:08.2684473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-08-17T12:08:13.2176370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-08-17T12:08:13.4279036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-08-17T12:08:13.6728015Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-08-17T12:08:14.4673851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-08-17T12:08:14.6763817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-08-17T12:08:15.3702109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-08-17T12:08:15.6556802Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-08-17T12:08:16.2238516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-08-17T12:08:18.2654623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-08-17T12:08:18.7484385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-08-17T12:08:20.9242020Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-08-17T12:08:20.9338525Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-08-17T12:08:20.9412282Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-08-17T12:08:20.9619653Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-08-17T12:08:20.9827081Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-08-17T12:08:21.5452053Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-08-17T12:08:21.5646817Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-08-17T12:08:21.6593931Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d' 2022-08-17T12:08:21.6912623Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-08-17T12:08:21.9621060Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8' 2022-08-17T12:08:22.1881011Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-08-17T12:08:22.2301718Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030' 2022-08-17T12:08:22.2315825Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:22.2317708Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:22.2319766Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:22.2328565Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:22.2344994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-08-17T12:08:23.2104723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-08-17T12:08:23.7251280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-08-17T12:08:24.7706955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-08-17T12:08:25.0896974Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-08-17T12:08:25.1843506Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-08-17T12:08:25.2390943Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-08-17T12:08:25.2483786Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-08-17T12:08:25.3299836Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-08-17T12:08:25.3618142Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2022-08-17T12:08:25.3700685Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-08-17T12:08:25.4078453Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-08-17T12:08:25.4300031Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72' 2022-08-17T12:08:25.4739960Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-08-17T12:08:25.4843827Z Submodule path 'third_party/ideep': checked out '8a114a51c116b55c4ceb689b98746786bd00c29b' 2022-08-17T12:08:25.4856073Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:25.4878187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-08-17T12:08:32.5291954Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46' 2022-08-17T12:08:32.5307398Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:32.5330393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-08-17T12:08:39.6178858Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815' 2022-08-17T12:08:39.6272509Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-08-17T12:08:39.6399506Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-08-17T12:08:39.7280697Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573' 2022-08-17T12:08:39.7294412Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:39.7296301Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:39.7319332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-08-17T12:08:40.9490249Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-08-17T12:08:42.0111290Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-08-17T12:08:42.0624204Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-08-17T12:08:42.0805245Z Submodule path 'third_party/nccl/nccl': checked out '19ab67d1727d337d10d0a48cbaf5cd119b8d88f1' 2022-08-17T12:08:42.0936183Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-08-17T12:08:42.1934992Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-08-17T12:08:42.4197842Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-08-17T12:08:42.4224727Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:42.4226354Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:42.4250405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-08-17T12:08:42.8011678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-08-17T12:08:43.6560730Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-08-17T12:08:43.6847666Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-08-17T12:08:43.6985556Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-08-17T12:08:43.6997969Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:43.7019117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-08-17T12:08:45.5357561Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-08-17T12:08:45.5375630Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:45.5377550Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:45.5400385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-08-17T12:08:45.9592147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-08-17T12:08:46.7724499Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-08-17T12:08:46.8343493Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-08-17T12:08:46.8357577Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:46.8380103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-08-17T12:08:47.0545526Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-08-17T12:08:47.0629558Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-08-17T12:08:47.3006670Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-08-17T12:08:47.3024841Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:47.3026600Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:47.3049734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-08-17T12:08:47.6724346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-08-17T12:08:48.7777978Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-08-17T12:08:48.8427641Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-08-17T12:08:48.8506906Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-08-17T12:08:48.8606745Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-08-17T12:08:48.8911471Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020' 2022-08-17T12:08:48.8991651Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-08-17T12:08:48.9256132Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-08-17T12:08:48.9342466Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-08-17T12:08:48.9754124Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-08-17T12:08:49.0769800Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-08-17T12:08:49.1010610Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-08-17T12:08:49.1023912Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:49.1025835Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:49.1028030Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:49.1030410Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:49.1053515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-08-17T12:08:50.2499137Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-08-17T12:08:50.5039155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-08-17T12:08:51.8328831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-08-17T12:08:52.6909019Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-08-17T12:08:52.7042525Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-08-17T12:08:52.7652086Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-08-17T12:08:52.7906387Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-08-17T12:08:52.7918843Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:52.7940059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-08-17T12:08:53.0277860Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-08-17T12:08:53.1535949Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-08-17T12:08:53.1563726Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-08-17T12:08:53.1822546Z Entering 'android/libs/fbjni' 2022-08-17T12:08:53.1856983Z Entering 'third_party/FP16' 2022-08-17T12:08:53.1891396Z Entering 'third_party/FXdiv' 2022-08-17T12:08:53.1925002Z Entering 'third_party/NNPACK' 2022-08-17T12:08:53.1959264Z Entering 'third_party/QNNPACK' 2022-08-17T12:08:53.1993544Z Entering 'third_party/XNNPACK' 2022-08-17T12:08:53.2038572Z Entering 'third_party/benchmark' 2022-08-17T12:08:53.2072857Z Entering 'third_party/cpuinfo' 2022-08-17T12:08:53.2106598Z Entering 'third_party/cub' 2022-08-17T12:08:53.2140419Z Entering 'third_party/cudnn_frontend' 2022-08-17T12:08:53.2179110Z Entering 'third_party/eigen' 2022-08-17T12:08:53.2214961Z Entering 'third_party/fbgemm' 2022-08-17T12:08:53.2250046Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:53.2282845Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:53.2316655Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:53.2349544Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:53.2385242Z Entering 'third_party/flatbuffers' 2022-08-17T12:08:53.2420315Z Entering 'third_party/fmt' 2022-08-17T12:08:53.2453574Z Entering 'third_party/foxi' 2022-08-17T12:08:53.2487593Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:08:53.2521292Z Entering 'third_party/gloo' 2022-08-17T12:08:53.2555338Z Entering 'third_party/googletest' 2022-08-17T12:08:53.2590534Z Entering 'third_party/ideep' 2022-08-17T12:08:53.2624315Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:53.2661433Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:53.2701386Z Entering 'third_party/ios-cmake' 2022-08-17T12:08:53.2735424Z Entering 'third_party/ittapi' 2022-08-17T12:08:53.2769192Z Entering 'third_party/kineto' 2022-08-17T12:08:53.2803528Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:53.2838815Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:53.2874568Z Entering 'third_party/nccl/nccl' 2022-08-17T12:08:53.2910066Z Entering 'third_party/neon2sse' 2022-08-17T12:08:53.2944587Z Entering 'third_party/nlohmann' 2022-08-17T12:08:53.2979899Z Entering 'third_party/onnx' 2022-08-17T12:08:53.3028750Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.3063849Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.3100147Z Entering 'third_party/onnx-tensorrt' 2022-08-17T12:08:53.3135676Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:53.3174486Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.3208742Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.3242261Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.3280230Z Entering 'third_party/pocketfft' 2022-08-17T12:08:53.3315582Z Entering 'third_party/protobuf' 2022-08-17T12:08:53.3352967Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:53.3386797Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:53.3421594Z Entering 'third_party/psimd' 2022-08-17T12:08:53.3455565Z Entering 'third_party/pthreadpool' 2022-08-17T12:08:53.3490364Z Entering 'third_party/pybind11' 2022-08-17T12:08:53.3524390Z Entering 'third_party/python-enum' 2022-08-17T12:08:53.3558728Z Entering 'third_party/python-peachpy' 2022-08-17T12:08:53.3591906Z Entering 'third_party/python-six' 2022-08-17T12:08:53.3625200Z Entering 'third_party/sleef' 2022-08-17T12:08:53.3658958Z Entering 'third_party/tbb' 2022-08-17T12:08:53.3693978Z Entering 'third_party/tensorpipe' 2022-08-17T12:08:53.3728684Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:53.3762222Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:53.3795647Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:53.3829165Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:53.3862017Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.3897553Z Entering 'third_party/zstd' 2022-08-17T12:08:53.3938876Z ##[endgroup] 2022-08-17T12:08:53.3939279Z ##[group]Persisting credentials for submodules 2022-08-17T12:08:53.3946235Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-08-17T12:08:53.4198277Z Entering 'android/libs/fbjni' 2022-08-17T12:08:53.4230001Z Entering 'third_party/FP16' 2022-08-17T12:08:53.4263305Z Entering 'third_party/FXdiv' 2022-08-17T12:08:53.4296573Z Entering 'third_party/NNPACK' 2022-08-17T12:08:53.4330166Z Entering 'third_party/QNNPACK' 2022-08-17T12:08:53.4363157Z Entering 'third_party/XNNPACK' 2022-08-17T12:08:53.4407477Z Entering 'third_party/benchmark' 2022-08-17T12:08:53.4440682Z Entering 'third_party/cpuinfo' 2022-08-17T12:08:53.4475013Z Entering 'third_party/cub' 2022-08-17T12:08:53.4508319Z Entering 'third_party/cudnn_frontend' 2022-08-17T12:08:53.4546963Z Entering 'third_party/eigen' 2022-08-17T12:08:53.4581795Z Entering 'third_party/fbgemm' 2022-08-17T12:08:53.4615294Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:53.4647647Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:53.4680201Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:53.4713249Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:53.4746684Z Entering 'third_party/flatbuffers' 2022-08-17T12:08:53.4782605Z Entering 'third_party/fmt' 2022-08-17T12:08:53.4815664Z Entering 'third_party/foxi' 2022-08-17T12:08:53.4849514Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:08:53.4883090Z Entering 'third_party/gloo' 2022-08-17T12:08:53.4916338Z Entering 'third_party/googletest' 2022-08-17T12:08:53.4949974Z Entering 'third_party/ideep' 2022-08-17T12:08:53.4983694Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:53.5017690Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:53.5056227Z Entering 'third_party/ios-cmake' 2022-08-17T12:08:53.5090598Z Entering 'third_party/ittapi' 2022-08-17T12:08:53.5124122Z Entering 'third_party/kineto' 2022-08-17T12:08:53.5158073Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:53.5191984Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:53.5228061Z Entering 'third_party/nccl/nccl' 2022-08-17T12:08:53.5261842Z Entering 'third_party/neon2sse' 2022-08-17T12:08:53.5295140Z Entering 'third_party/nlohmann' 2022-08-17T12:08:53.5330809Z Entering 'third_party/onnx' 2022-08-17T12:08:53.5375465Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.5408959Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.5443786Z Entering 'third_party/onnx-tensorrt' 2022-08-17T12:08:53.5476621Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:53.5514948Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.5549683Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.5582869Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.5620421Z Entering 'third_party/pocketfft' 2022-08-17T12:08:53.5653995Z Entering 'third_party/protobuf' 2022-08-17T12:08:53.5691870Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:53.5725232Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:53.5760215Z Entering 'third_party/psimd' 2022-08-17T12:08:53.5793703Z Entering 'third_party/pthreadpool' 2022-08-17T12:08:53.5827079Z Entering 'third_party/pybind11' 2022-08-17T12:08:53.5862862Z Entering 'third_party/python-enum' 2022-08-17T12:08:53.5896688Z Entering 'third_party/python-peachpy' 2022-08-17T12:08:53.5930137Z Entering 'third_party/python-six' 2022-08-17T12:08:53.5963642Z Entering 'third_party/sleef' 2022-08-17T12:08:53.5997295Z Entering 'third_party/tbb' 2022-08-17T12:08:53.6033028Z Entering 'third_party/tensorpipe' 2022-08-17T12:08:53.6066812Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:53.6100656Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:53.6134154Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:53.6167414Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:53.6199245Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.6234913Z Entering 'third_party/zstd' 2022-08-17T12:08:53.6280719Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-08-17T12:08:53.6532259Z Entering 'android/libs/fbjni' 2022-08-17T12:08:53.6562853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-08-17T12:08:53.6577469Z Entering 'third_party/FP16' 2022-08-17T12:08:53.6609291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-08-17T12:08:53.6622883Z Entering 'third_party/FXdiv' 2022-08-17T12:08:53.6655861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-08-17T12:08:53.6669469Z Entering 'third_party/NNPACK' 2022-08-17T12:08:53.6701729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-08-17T12:08:53.6715579Z Entering 'third_party/QNNPACK' 2022-08-17T12:08:53.6747594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-08-17T12:08:53.6762015Z Entering 'third_party/XNNPACK' 2022-08-17T12:08:53.6793526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-08-17T12:08:53.6817890Z Entering 'third_party/benchmark' 2022-08-17T12:08:53.6850746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-08-17T12:08:53.6864392Z Entering 'third_party/cpuinfo' 2022-08-17T12:08:53.6895866Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-08-17T12:08:53.6909868Z Entering 'third_party/cub' 2022-08-17T12:08:53.6941929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-08-17T12:08:53.6956175Z Entering 'third_party/cudnn_frontend' 2022-08-17T12:08:53.6988170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-08-17T12:08:53.7008271Z Entering 'third_party/eigen' 2022-08-17T12:08:53.7039516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-08-17T12:08:53.7055771Z Entering 'third_party/fbgemm' 2022-08-17T12:08:53.7087628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-08-17T12:08:53.7101622Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:53.7133856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-08-17T12:08:53.7147229Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:53.7178821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-08-17T12:08:53.7192488Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:53.7224302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-08-17T12:08:53.7237641Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:53.7268879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-08-17T12:08:53.7284754Z Entering 'third_party/flatbuffers' 2022-08-17T12:08:53.7317349Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-08-17T12:08:53.7333733Z Entering 'third_party/fmt' 2022-08-17T12:08:53.7366182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-08-17T12:08:53.7379989Z Entering 'third_party/foxi' 2022-08-17T12:08:53.7411580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-08-17T12:08:53.7425470Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:08:53.7457265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-08-17T12:08:53.7472167Z Entering 'third_party/gloo' 2022-08-17T12:08:53.7503975Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-08-17T12:08:53.7518677Z Entering 'third_party/googletest' 2022-08-17T12:08:53.7551155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-08-17T12:08:53.7565734Z Entering 'third_party/ideep' 2022-08-17T12:08:53.7596888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-08-17T12:08:53.7610501Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:53.7641484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-08-17T12:08:53.7656742Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:53.7688494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-08-17T12:08:53.7707802Z Entering 'third_party/ios-cmake' 2022-08-17T12:08:53.7741062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-08-17T12:08:53.7755178Z Entering 'third_party/ittapi' 2022-08-17T12:08:53.7786527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-08-17T12:08:53.7800497Z Entering 'third_party/kineto' 2022-08-17T12:08:53.7832325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-08-17T12:08:53.7847073Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:53.7878303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-08-17T12:08:53.7892244Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:53.7924094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-08-17T12:08:53.7939397Z Entering 'third_party/nccl/nccl' 2022-08-17T12:08:53.7970552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-08-17T12:08:53.7984150Z Entering 'third_party/neon2sse' 2022-08-17T12:08:53.8016142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-08-17T12:08:53.8028873Z Entering 'third_party/nlohmann' 2022-08-17T12:08:53.8061448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-08-17T12:08:53.8076447Z Entering 'third_party/onnx' 2022-08-17T12:08:53.8108702Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-08-17T12:08:53.8134227Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.8166407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-08-17T12:08:53.8182094Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.8213309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-08-17T12:08:53.8228263Z Entering 'third_party/onnx-tensorrt' 2022-08-17T12:08:53.8259932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-08-17T12:08:53.8273269Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:53.8305580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-08-17T12:08:53.8324381Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:53.8356008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-08-17T12:08:53.8371111Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:53.8401818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-08-17T12:08:53.8415163Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.8446638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-08-17T12:08:53.8463980Z Entering 'third_party/pocketfft' 2022-08-17T12:08:53.8495813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-08-17T12:08:53.8509349Z Entering 'third_party/protobuf' 2022-08-17T12:08:53.8542149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-08-17T12:08:53.8560154Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:53.8592417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-08-17T12:08:53.8606162Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:53.8638261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-08-17T12:08:53.8654226Z Entering 'third_party/psimd' 2022-08-17T12:08:53.8686264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-08-17T12:08:53.8699924Z Entering 'third_party/pthreadpool' 2022-08-17T12:08:53.8733655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-08-17T12:08:53.8747691Z Entering 'third_party/pybind11' 2022-08-17T12:08:53.8779390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-08-17T12:08:53.8794260Z Entering 'third_party/python-enum' 2022-08-17T12:08:53.8826177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-08-17T12:08:53.8840141Z Entering 'third_party/python-peachpy' 2022-08-17T12:08:53.8871148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-08-17T12:08:53.8885609Z Entering 'third_party/python-six' 2022-08-17T12:08:53.8917327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-08-17T12:08:53.8931552Z Entering 'third_party/sleef' 2022-08-17T12:08:53.8963167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-08-17T12:08:53.8978361Z Entering 'third_party/tbb' 2022-08-17T12:08:53.9011549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-08-17T12:08:53.9028000Z Entering 'third_party/tensorpipe' 2022-08-17T12:08:53.9062776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-08-17T12:08:53.9077045Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:53.9109755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-08-17T12:08:53.9123746Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:53.9155327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-08-17T12:08:53.9170222Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:53.9200965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-08-17T12:08:53.9215057Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:53.9246195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-08-17T12:08:53.9259418Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:53.9290820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-08-17T12:08:53.9307749Z Entering 'third_party/zstd' 2022-08-17T12:08:53.9339421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-08-17T12:08:54.0241305Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-08-17T12:08:54.0513578Z Entering 'android/libs/fbjni' 2022-08-17T12:08:54.0550171Z Entering 'third_party/FP16' 2022-08-17T12:08:54.0589477Z Entering 'third_party/FXdiv' 2022-08-17T12:08:54.0628438Z Entering 'third_party/NNPACK' 2022-08-17T12:08:54.0664809Z Entering 'third_party/QNNPACK' 2022-08-17T12:08:54.0700167Z Entering 'third_party/XNNPACK' 2022-08-17T12:08:54.0746935Z Entering 'third_party/benchmark' 2022-08-17T12:08:54.0784703Z Entering 'third_party/cpuinfo' 2022-08-17T12:08:54.0821580Z Entering 'third_party/cub' 2022-08-17T12:08:54.0857054Z Entering 'third_party/cudnn_frontend' 2022-08-17T12:08:54.0897544Z Entering 'third_party/eigen' 2022-08-17T12:08:54.0935243Z Entering 'third_party/fbgemm' 2022-08-17T12:08:54.0971045Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:54.1005284Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:54.1040604Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:54.1075485Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:54.1111761Z Entering 'third_party/flatbuffers' 2022-08-17T12:08:54.1151106Z Entering 'third_party/fmt' 2022-08-17T12:08:54.1186825Z Entering 'third_party/foxi' 2022-08-17T12:08:54.1222708Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:08:54.1259473Z Entering 'third_party/gloo' 2022-08-17T12:08:54.1295319Z Entering 'third_party/googletest' 2022-08-17T12:08:54.1329676Z Entering 'third_party/ideep' 2022-08-17T12:08:54.1363474Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:54.1398439Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:54.1438664Z Entering 'third_party/ios-cmake' 2022-08-17T12:08:54.1473077Z Entering 'third_party/ittapi' 2022-08-17T12:08:54.1508399Z Entering 'third_party/kineto' 2022-08-17T12:08:54.1542641Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:54.1576260Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:54.1611079Z Entering 'third_party/nccl/nccl' 2022-08-17T12:08:54.1645200Z Entering 'third_party/neon2sse' 2022-08-17T12:08:54.1678989Z Entering 'third_party/nlohmann' 2022-08-17T12:08:54.1716189Z Entering 'third_party/onnx' 2022-08-17T12:08:54.1763369Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:54.1797547Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:54.1832325Z Entering 'third_party/onnx-tensorrt' 2022-08-17T12:08:54.1867031Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:54.1904800Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:54.1940396Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:54.1973495Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:54.2012057Z Entering 'third_party/pocketfft' 2022-08-17T12:08:54.2046620Z Entering 'third_party/protobuf' 2022-08-17T12:08:54.2084078Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:54.2119619Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:54.2155359Z Entering 'third_party/psimd' 2022-08-17T12:08:54.2189807Z Entering 'third_party/pthreadpool' 2022-08-17T12:08:54.2224254Z Entering 'third_party/pybind11' 2022-08-17T12:08:54.2257578Z Entering 'third_party/python-enum' 2022-08-17T12:08:54.2291968Z Entering 'third_party/python-peachpy' 2022-08-17T12:08:54.2327708Z Entering 'third_party/python-six' 2022-08-17T12:08:54.2362263Z Entering 'third_party/sleef' 2022-08-17T12:08:54.2397136Z Entering 'third_party/tbb' 2022-08-17T12:08:54.2433661Z Entering 'third_party/tensorpipe' 2022-08-17T12:08:54.2469674Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:54.2504585Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:54.2539669Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:54.2574280Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:54.2609329Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:54.2645735Z Entering 'third_party/zstd' 2022-08-17T12:08:54.2693562Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-08-17T12:08:54.2946399Z Entering 'android/libs/fbjni' 2022-08-17T12:08:54.2980824Z Entering 'third_party/FP16' 2022-08-17T12:08:54.3015914Z Entering 'third_party/FXdiv' 2022-08-17T12:08:54.3049834Z Entering 'third_party/NNPACK' 2022-08-17T12:08:54.3085379Z Entering 'third_party/QNNPACK' 2022-08-17T12:08:54.3120615Z Entering 'third_party/XNNPACK' 2022-08-17T12:08:54.3164847Z Entering 'third_party/benchmark' 2022-08-17T12:08:54.3199232Z Entering 'third_party/cpuinfo' 2022-08-17T12:08:54.3235553Z Entering 'third_party/cub' 2022-08-17T12:08:54.3272599Z Entering 'third_party/cudnn_frontend' 2022-08-17T12:08:54.3311872Z Entering 'third_party/eigen' 2022-08-17T12:08:54.3348200Z Entering 'third_party/fbgemm' 2022-08-17T12:08:54.3381926Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T12:08:54.3416358Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T12:08:54.3450187Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T12:08:54.3484286Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T12:08:54.3518890Z Entering 'third_party/flatbuffers' 2022-08-17T12:08:54.3555373Z Entering 'third_party/fmt' 2022-08-17T12:08:54.3589036Z Entering 'third_party/foxi' 2022-08-17T12:08:54.3622833Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T12:08:54.3658168Z Entering 'third_party/gloo' 2022-08-17T12:08:54.3691880Z Entering 'third_party/googletest' 2022-08-17T12:08:54.3725124Z Entering 'third_party/ideep' 2022-08-17T12:08:54.3758847Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T12:08:54.3794827Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T12:08:54.3833116Z Entering 'third_party/ios-cmake' 2022-08-17T12:08:54.3868261Z Entering 'third_party/ittapi' 2022-08-17T12:08:54.3903035Z Entering 'third_party/kineto' 2022-08-17T12:08:54.3937746Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T12:08:54.3971711Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T12:08:54.4005780Z Entering 'third_party/nccl/nccl' 2022-08-17T12:08:54.4039769Z Entering 'third_party/neon2sse' 2022-08-17T12:08:54.4073503Z Entering 'third_party/nlohmann' 2022-08-17T12:08:54.4108697Z Entering 'third_party/onnx' 2022-08-17T12:08:54.4154849Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T12:08:54.4189252Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T12:08:54.4224268Z Entering 'third_party/onnx-tensorrt' 2022-08-17T12:08:54.4257584Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T12:08:54.4296892Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T12:08:54.4329813Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T12:08:54.4362942Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T12:08:54.4403613Z Entering 'third_party/pocketfft' 2022-08-17T12:08:54.4438455Z Entering 'third_party/protobuf' 2022-08-17T12:08:54.4476040Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T12:08:54.4509970Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T12:08:54.4546330Z Entering 'third_party/psimd' 2022-08-17T12:08:54.4580593Z Entering 'third_party/pthreadpool' 2022-08-17T12:08:54.4614463Z Entering 'third_party/pybind11' 2022-08-17T12:08:54.4648464Z Entering 'third_party/python-enum' 2022-08-17T12:08:54.4682122Z Entering 'third_party/python-peachpy' 2022-08-17T12:08:54.4715804Z Entering 'third_party/python-six' 2022-08-17T12:08:54.4749268Z Entering 'third_party/sleef' 2022-08-17T12:08:54.4782642Z Entering 'third_party/tbb' 2022-08-17T12:08:54.4817630Z Entering 'third_party/tensorpipe' 2022-08-17T12:08:54.4852186Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T12:08:54.4885447Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T12:08:54.4918178Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T12:08:54.4952068Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T12:08:54.4984926Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T12:08:54.5020318Z Entering 'third_party/zstd' 2022-08-17T12:08:54.5063721Z ##[endgroup] 2022-08-17T12:08:54.5100951Z [command]/usr/bin/git log -1 --format='%H' 2022-08-17T12:08:54.5125615Z 'ce6a3c605df99d1df57c0dda75c06d748e54ed2a' 2022-08-17T12:08:54.5247624Z Prepare all required actions 2022-08-17T12:08:54.5298507Z ##[group]Run ./.github/actions/setup-linux 2022-08-17T12:08:54.5298711Z env: 2022-08-17T12:08:54.5298891Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:54.5299071Z ##[endgroup] 2022-08-17T12:08:54.5342405Z ##[group]Run set -euo pipefail 2022-08-17T12:08:54.5342650Z set -euo pipefail 2022-08-17T12:08:54.5342869Z function get_ec2_metadata() { 2022-08-17T12:08:54.5343114Z  # Pulled from instance metadata endpoint for EC2 2022-08-17T12:08:54.5343489Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2022-08-17T12:08:54.5343801Z  category=$1 2022-08-17T12:08:54.5344055Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2022-08-17T12:08:54.5344276Z } 2022-08-17T12:08:54.5344485Z echo "ami-id: $(get_ec2_metadata ami-id)" 2022-08-17T12:08:54.5344785Z echo "instance-id: $(get_ec2_metadata instance-id)" 2022-08-17T12:08:54.5345069Z echo "instance-type: $(get_ec2_metadata instance-type)" 2022-08-17T12:08:54.5345331Z echo "system info $(uname -a)" 2022-08-17T12:08:54.5357192Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:08:54.5357412Z env: 2022-08-17T12:08:54.5357596Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:54.5357794Z ##[endgroup] 2022-08-17T12:08:54.5442906Z ami-id: ami-096198a0bccc6bad4 2022-08-17T12:08:54.5494403Z instance-id: i-068af22e3918a20a1 2022-08-17T12:08:54.5543695Z instance-type: c5.2xlarge 2022-08-17T12:08:54.5550550Z system info Linux ip-10-0-2-81.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2022-08-17T12:08:54.5567102Z ##[group]Run if systemctl is-active --quiet docker; then 2022-08-17T12:08:54.5567396Z if systemctl is-active --quiet docker; then 2022-08-17T12:08:54.5567651Z  echo "Docker daemon is running..."; 2022-08-17T12:08:54.5567852Z else 2022-08-17T12:08:54.5568098Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2022-08-17T12:08:54.5568333Z fi 2022-08-17T12:08:54.5578673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:08:54.5578901Z env: 2022-08-17T12:08:54.5579079Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:54.5579261Z ##[endgroup] 2022-08-17T12:08:54.5620443Z Docker daemon is running... 2022-08-17T12:08:54.5636305Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-08-17T12:08:54.5636676Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-08-17T12:08:54.5636978Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-08-17T12:08:54.5637478Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2022-08-17T12:08:54.5637833Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2022-08-17T12:08:54.5648286Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:08:54.5648516Z env: 2022-08-17T12:08:54.5648688Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:54.5648893Z AWS_RETRY_MODE: standard 2022-08-17T12:08:54.5649094Z AWS_MAX_ATTEMPTS: 5 2022-08-17T12:08:54.5649288Z AWS_DEFAULT_REGION: us-east-1 2022-08-17T12:08:54.5649489Z ##[endgroup] 2022-08-17T12:08:55.3330470Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2022-08-17T12:08:55.3330911Z Configure a credential helper to remove this warning. See 2022-08-17T12:08:55.3332910Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2022-08-17T12:08:55.3333351Z 2022-08-17T12:08:55.3333470Z Login Succeeded 2022-08-17T12:08:55.3369682Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-08-17T12:08:55.3370004Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-08-17T12:08:55.3370358Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-08-17T12:08:55.3381795Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:08:55.3382025Z env: 2022-08-17T12:08:55.3382209Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:55.3382394Z ##[endgroup] 2022-08-17T12:08:55.3443853Z Prepare all required actions 2022-08-17T12:08:55.3444117Z Getting action download info 2022-08-17T12:08:55.6143465Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638) 2022-08-17T12:08:55.7346790Z ##[group]Run ./.github/actions/setup-ssh 2022-08-17T12:08:55.7347007Z with: 2022-08-17T12:08:55.7347391Z github-secret: *** 2022-08-17T12:08:55.7347574Z env: 2022-08-17T12:08:55.7347757Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:55.7347939Z ##[endgroup] 2022-08-17T12:08:55.7369371Z ##[group]Run seemethere/add-github-ssh-key@v1 2022-08-17T12:08:55.7369637Z with: 2022-08-17T12:08:55.7370002Z GITHUB_TOKEN: *** 2022-08-17T12:08:55.7370195Z activate-with-label: false 2022-08-17T12:08:55.7370412Z label: with-ssh 2022-08-17T12:08:55.7370617Z remove-existing-keys: true 2022-08-17T12:08:55.7370797Z env: 2022-08-17T12:08:55.7370977Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:55.7371174Z ##[endgroup] 2022-08-17T12:08:56.1432674Z Grabbing public ssh keys from https://github.com/IvanYashchuk.keys 2022-08-17T12:08:56.2256851Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2022-08-17T12:08:56.2269960Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2022-08-17T12:08:56.2293117Z Login using: ssh ec2-user@ec2-3-86-221-63.compute-1.amazonaws.com 2022-08-17T12:08:56.2339291Z Prepare all required actions 2022-08-17T12:08:56.2358494Z ##[group]Run ./.github/actions/pull-docker-image 2022-08-17T12:08:56.2358713Z with: 2022-08-17T12:08:56.2359061Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:08:56.2359379Z env: 2022-08-17T12:08:56.2359567Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:56.2359762Z ##[endgroup] 2022-08-17T12:08:56.2373517Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-08-17T12:08:56.2373803Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-08-17T12:08:56.2374082Z # ignore output since only exit code is used for conditional 2022-08-17T12:08:56.2374364Z # only pull docker image if it's not available locally 2022-08-17T12:08:56.2374675Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-08-17T12:08:56.2374990Z  retry docker pull "${DOCKER_IMAGE}" 2022-08-17T12:08:56.2375182Z fi 2022-08-17T12:08:56.2386131Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:08:56.2386359Z env: 2022-08-17T12:08:56.2386530Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:08:56.2386900Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:08:56.2387248Z ##[endgroup] 2022-08-17T12:08:56.4527139Z a347f7e7645f04fc68e4f87c73cf0385233153b8: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7 2022-08-17T12:08:56.4527826Z 3b65ec22a9e9: Pulling fs layer 2022-08-17T12:08:56.4528284Z ef46a21e4004: Pulling fs layer 2022-08-17T12:08:56.4568505Z 0cd966808131: Pulling fs layer 2022-08-17T12:08:56.4568873Z d96ffce08e66: Pulling fs layer 2022-08-17T12:08:56.4569182Z 6c71e48d181e: Pulling fs layer 2022-08-17T12:08:56.4569509Z 3e0259f02183: Pulling fs layer 2022-08-17T12:08:56.4569861Z 9dc5c789901b: Pulling fs layer 2022-08-17T12:08:56.4570180Z 4ec60a8f4b0d: Pulling fs layer 2022-08-17T12:08:56.4570484Z efb8d44a1237: Pulling fs layer 2022-08-17T12:08:56.4570799Z 0846fd5f50b1: Pulling fs layer 2022-08-17T12:08:56.4571034Z 2f02b9c38fdd: Pulling fs layer 2022-08-17T12:08:56.4571228Z 5e6332363e8c: Pulling fs layer 2022-08-17T12:08:56.4571425Z 7864323978cb: Pulling fs layer 2022-08-17T12:08:56.4571781Z 6c71e48d181e: Waiting 2022-08-17T12:08:56.4571970Z 4ce5b8109bcb: Pulling fs layer 2022-08-17T12:08:56.4572158Z 9a5798f00c55: Pulling fs layer 2022-08-17T12:08:56.4572341Z b477457c93ae: Pulling fs layer 2022-08-17T12:08:56.4572533Z b8835462091e: Pulling fs layer 2022-08-17T12:08:56.4572718Z 3e0259f02183: Waiting 2022-08-17T12:08:56.4572880Z 9dc5c789901b: Waiting 2022-08-17T12:08:56.4573072Z 5c88d6b28819: Pulling fs layer 2022-08-17T12:08:56.4573275Z 8b38f067771b: Pulling fs layer 2022-08-17T12:08:56.4573469Z b5dbbce4b04e: Pulling fs layer 2022-08-17T12:08:56.4573663Z 4ec60a8f4b0d: Waiting 2022-08-17T12:08:56.4573845Z 00a3d832f66e: Pulling fs layer 2022-08-17T12:08:56.4574028Z 682b682d9e93: Pulling fs layer 2022-08-17T12:08:56.4574199Z efb8d44a1237: Waiting 2022-08-17T12:08:56.4574377Z f62d6a29ee25: Pulling fs layer 2022-08-17T12:08:56.4574550Z 0846fd5f50b1: Waiting 2022-08-17T12:08:56.4574707Z 5e6332363e8c: Waiting 2022-08-17T12:08:56.4574870Z d96ffce08e66: Waiting 2022-08-17T12:08:56.4575037Z 2f02b9c38fdd: Waiting 2022-08-17T12:08:56.4575195Z 7864323978cb: Waiting 2022-08-17T12:08:56.4575371Z 9f2aa5b7f4ee: Pulling fs layer 2022-08-17T12:08:56.4575559Z c1cc323d4a98: Pulling fs layer 2022-08-17T12:08:56.4575743Z d323eef1e7e1: Pulling fs layer 2022-08-17T12:08:56.4575940Z ade3e8c67ae6: Pulling fs layer 2022-08-17T12:08:56.4576131Z a492728f4507: Pulling fs layer 2022-08-17T12:08:56.4576298Z 00a3d832f66e: Waiting 2022-08-17T12:08:56.4576464Z 9f2aa5b7f4ee: Waiting 2022-08-17T12:08:56.4576644Z fd9ccd86a681: Pulling fs layer 2022-08-17T12:08:56.4576818Z b477457c93ae: Waiting 2022-08-17T12:08:56.4576994Z d932c4ca853a: Pulling fs layer 2022-08-17T12:08:56.4577234Z 682b682d9e93: Waiting 2022-08-17T12:08:56.4577407Z 15c8440a96aa: Pulling fs layer 2022-08-17T12:08:56.4577583Z b8835462091e: Waiting 2022-08-17T12:08:56.4577747Z c1cc323d4a98: Waiting 2022-08-17T12:08:56.4577913Z 337a233ac028: Pulling fs layer 2022-08-17T12:08:56.4578082Z 9a5798f00c55: Waiting 2022-08-17T12:08:56.4578285Z 97be5b091dd9: Pulling fs layer 2022-08-17T12:08:56.4578472Z 5c88d6b28819: Waiting 2022-08-17T12:08:56.4578646Z 2ada12616b22: Pulling fs layer 2022-08-17T12:08:56.4578837Z e69a2cbb6fd4: Pulling fs layer 2022-08-17T12:08:56.4579012Z a492728f4507: Waiting 2022-08-17T12:08:56.4579201Z 0044ee4ac606: Pulling fs layer 2022-08-17T12:08:56.4579385Z 8b38f067771b: Waiting 2022-08-17T12:08:56.4579552Z 15c8440a96aa: Waiting 2022-08-17T12:08:56.4579733Z fd9ccd86a681: Waiting 2022-08-17T12:08:56.4579942Z b5dbbce4b04e: Waiting 2022-08-17T12:08:56.4580110Z 4ce5b8109bcb: Waiting 2022-08-17T12:08:56.4580295Z e601349e520b: Pulling fs layer 2022-08-17T12:08:56.4580528Z 2ada12616b22: Waiting 2022-08-17T12:08:56.4580711Z 337a233ac028: Waiting 2022-08-17T12:08:56.4580888Z 97be5b091dd9: Waiting 2022-08-17T12:08:56.4581085Z 0044ee4ac606: Waiting 2022-08-17T12:08:56.4581261Z ade3e8c67ae6: Waiting 2022-08-17T12:08:56.4581456Z 591a21ed2ab8: Pulling fs layer 2022-08-17T12:08:56.4581650Z d932c4ca853a: Waiting 2022-08-17T12:08:56.4581815Z f62d6a29ee25: Waiting 2022-08-17T12:08:56.4582009Z e4521e0c3854: Pulling fs layer 2022-08-17T12:08:56.4582198Z e601349e520b: Waiting 2022-08-17T12:08:56.4582373Z f568ee9f41e3: Pulling fs layer 2022-08-17T12:08:56.4582578Z 19df37b4c74c: Pulling fs layer 2022-08-17T12:08:56.4582770Z e4521e0c3854: Waiting 2022-08-17T12:08:56.4582960Z c363a482236a: Pulling fs layer 2022-08-17T12:08:56.4583142Z e69a2cbb6fd4: Waiting 2022-08-17T12:08:56.4583338Z dcf43507e80e: Pulling fs layer 2022-08-17T12:08:56.4583548Z cdeff5e8900e: Pulling fs layer 2022-08-17T12:08:56.4583743Z 432ead746a51: Pulling fs layer 2022-08-17T12:08:56.4583933Z c363a482236a: Waiting 2022-08-17T12:08:56.4584116Z f568ee9f41e3: Waiting 2022-08-17T12:08:56.4584280Z 19df37b4c74c: Waiting 2022-08-17T12:08:56.4584470Z 7887f902330e: Pulling fs layer 2022-08-17T12:08:56.4584662Z dcf43507e80e: Waiting 2022-08-17T12:08:56.4584828Z cdeff5e8900e: Waiting 2022-08-17T12:08:56.4585020Z 0107d101d703: Pulling fs layer 2022-08-17T12:08:56.4585207Z 7887f902330e: Waiting 2022-08-17T12:08:56.4585380Z 424513ba624b: Pulling fs layer 2022-08-17T12:08:56.4585633Z d9c9fd17b60c: Pulling fs layer 2022-08-17T12:08:56.4585845Z aa1eabed864f: Pulling fs layer 2022-08-17T12:08:56.4586036Z c483592d5074: Pulling fs layer 2022-08-17T12:08:56.4586237Z 07f2aa52159a: Pulling fs layer 2022-08-17T12:08:56.4586442Z 13f262da6a6c: Pulling fs layer 2022-08-17T12:08:56.4586623Z aa1eabed864f: Waiting 2022-08-17T12:08:56.4586807Z 0107d101d703: Waiting 2022-08-17T12:08:56.4586983Z 424513ba624b: Waiting 2022-08-17T12:08:56.4587157Z 1e128c6311be: Pulling fs layer 2022-08-17T12:08:56.4587358Z 0a364ea25255: Pulling fs layer 2022-08-17T12:08:56.4587551Z 07f2aa52159a: Waiting 2022-08-17T12:08:56.4587715Z c483592d5074: Waiting 2022-08-17T12:08:56.4587890Z 0a364ea25255: Waiting 2022-08-17T12:08:56.4588078Z 9b8f9a96d4e8: Pulling fs layer 2022-08-17T12:08:56.4588264Z 708682444671: Pulling fs layer 2022-08-17T12:08:56.4588452Z d9c9fd17b60c: Waiting 2022-08-17T12:08:56.4588643Z 589913fd2607: Pulling fs layer 2022-08-17T12:08:56.4588832Z 7b3f9f48221a: Pulling fs layer 2022-08-17T12:08:56.4589022Z 708682444671: Waiting 2022-08-17T12:08:56.4589200Z 589913fd2607: Waiting 2022-08-17T12:08:56.5202789Z ef46a21e4004: Verifying Checksum 2022-08-17T12:08:56.5203049Z ef46a21e4004: Download complete 2022-08-17T12:08:56.5990189Z d96ffce08e66: Verifying Checksum 2022-08-17T12:08:56.5990588Z d96ffce08e66: Download complete 2022-08-17T12:08:56.6681062Z 6c71e48d181e: Verifying Checksum 2022-08-17T12:08:56.6681481Z 6c71e48d181e: Download complete 2022-08-17T12:08:56.7598776Z 3e0259f02183: Verifying Checksum 2022-08-17T12:08:56.7599036Z 3e0259f02183: Download complete 2022-08-17T12:08:56.8136949Z 3b65ec22a9e9: Verifying Checksum 2022-08-17T12:08:56.8137406Z 3b65ec22a9e9: Download complete 2022-08-17T12:08:56.8294940Z 9dc5c789901b: Download complete 2022-08-17T12:08:56.8910231Z 4ec60a8f4b0d: Download complete 2022-08-17T12:08:56.8970376Z efb8d44a1237: Verifying Checksum 2022-08-17T12:08:56.8970899Z efb8d44a1237: Download complete 2022-08-17T12:08:56.9636933Z 0846fd5f50b1: Download complete 2022-08-17T12:08:57.0404263Z 5e6332363e8c: Verifying Checksum 2022-08-17T12:08:57.0404672Z 5e6332363e8c: Download complete 2022-08-17T12:08:57.1209782Z 7864323978cb: Verifying Checksum 2022-08-17T12:08:57.1212791Z 7864323978cb: Download complete 2022-08-17T12:08:57.5750468Z 3b65ec22a9e9: Pull complete 2022-08-17T12:08:57.6827291Z ef46a21e4004: Pull complete 2022-08-17T12:08:57.7729078Z 2f02b9c38fdd: Verifying Checksum 2022-08-17T12:08:57.7729504Z 2f02b9c38fdd: Download complete 2022-08-17T12:08:57.8324576Z 9a5798f00c55: Verifying Checksum 2022-08-17T12:08:57.8324979Z 9a5798f00c55: Download complete 2022-08-17T12:08:57.8956064Z b477457c93ae: Verifying Checksum 2022-08-17T12:08:57.8956321Z b477457c93ae: Download complete 2022-08-17T12:08:58.3729357Z b8835462091e: Verifying Checksum 2022-08-17T12:08:58.3729738Z b8835462091e: Download complete 2022-08-17T12:08:58.4468827Z 5c88d6b28819: Verifying Checksum 2022-08-17T12:08:58.4469340Z 5c88d6b28819: Download complete 2022-08-17T12:08:58.5568908Z 8b38f067771b: Verifying Checksum 2022-08-17T12:08:58.5569423Z 8b38f067771b: Download complete 2022-08-17T12:08:58.6641470Z b5dbbce4b04e: Download complete 2022-08-17T12:08:58.7489364Z 00a3d832f66e: Verifying Checksum 2022-08-17T12:08:58.7489758Z 00a3d832f66e: Download complete 2022-08-17T12:08:58.8721062Z 682b682d9e93: Verifying Checksum 2022-08-17T12:08:58.8721445Z 682b682d9e93: Download complete 2022-08-17T12:08:59.1338349Z f62d6a29ee25: Verifying Checksum 2022-08-17T12:08:59.1338651Z f62d6a29ee25: Download complete 2022-08-17T12:08:59.2603630Z 9f2aa5b7f4ee: Download complete 2022-08-17T12:08:59.3430244Z c1cc323d4a98: Verifying Checksum 2022-08-17T12:08:59.3430667Z c1cc323d4a98: Download complete 2022-08-17T12:08:59.5688613Z 0cd966808131: Verifying Checksum 2022-08-17T12:08:59.5689095Z 0cd966808131: Download complete 2022-08-17T12:08:59.7121998Z ade3e8c67ae6: Verifying Checksum 2022-08-17T12:08:59.7122433Z ade3e8c67ae6: Download complete 2022-08-17T12:08:59.7976077Z a492728f4507: Verifying Checksum 2022-08-17T12:08:59.7976631Z a492728f4507: Download complete 2022-08-17T12:08:59.9331102Z fd9ccd86a681: Verifying Checksum 2022-08-17T12:08:59.9331448Z fd9ccd86a681: Download complete 2022-08-17T12:09:00.1353859Z d932c4ca853a: Verifying Checksum 2022-08-17T12:09:00.1354287Z d932c4ca853a: Download complete 2022-08-17T12:09:00.2299432Z 15c8440a96aa: Verifying Checksum 2022-08-17T12:09:00.2299827Z 15c8440a96aa: Download complete 2022-08-17T12:09:00.3104099Z d323eef1e7e1: Verifying Checksum 2022-08-17T12:09:00.3105520Z d323eef1e7e1: Download complete 2022-08-17T12:09:00.3848318Z 97be5b091dd9: Verifying Checksum 2022-08-17T12:09:00.3848834Z 97be5b091dd9: Download complete 2022-08-17T12:09:00.4812216Z 2ada12616b22: Verifying Checksum 2022-08-17T12:09:00.4812800Z 2ada12616b22: Download complete 2022-08-17T12:09:00.5567630Z e69a2cbb6fd4: Download complete 2022-08-17T12:09:00.6336157Z 0044ee4ac606: Verifying Checksum 2022-08-17T12:09:00.6338622Z 0044ee4ac606: Download complete 2022-08-17T12:09:00.7629096Z e601349e520b: Verifying Checksum 2022-08-17T12:09:00.7629566Z e601349e520b: Download complete 2022-08-17T12:09:00.8823919Z 591a21ed2ab8: Verifying Checksum 2022-08-17T12:09:00.8824612Z 591a21ed2ab8: Download complete 2022-08-17T12:09:01.0032988Z e4521e0c3854: Verifying Checksum 2022-08-17T12:09:01.0033428Z e4521e0c3854: Download complete 2022-08-17T12:09:01.0864325Z f568ee9f41e3: Verifying Checksum 2022-08-17T12:09:01.0883447Z f568ee9f41e3: Download complete 2022-08-17T12:09:01.1711814Z 19df37b4c74c: Verifying Checksum 2022-08-17T12:09:01.1712253Z 19df37b4c74c: Download complete 2022-08-17T12:09:01.2769514Z c363a482236a: Verifying Checksum 2022-08-17T12:09:01.2770113Z c363a482236a: Download complete 2022-08-17T12:09:01.3681552Z dcf43507e80e: Verifying Checksum 2022-08-17T12:09:01.3682776Z dcf43507e80e: Download complete 2022-08-17T12:09:01.4677877Z cdeff5e8900e: Verifying Checksum 2022-08-17T12:09:01.4678401Z cdeff5e8900e: Download complete 2022-08-17T12:09:01.9344213Z 432ead746a51: Verifying Checksum 2022-08-17T12:09:01.9344562Z 432ead746a51: Download complete 2022-08-17T12:09:02.0209166Z 7887f902330e: Verifying Checksum 2022-08-17T12:09:02.0210363Z 7887f902330e: Download complete 2022-08-17T12:09:02.1073285Z 0107d101d703: Download complete 2022-08-17T12:09:02.2112815Z 424513ba624b: Verifying Checksum 2022-08-17T12:09:02.2114005Z 424513ba624b: Download complete 2022-08-17T12:09:02.2886654Z d9c9fd17b60c: Verifying Checksum 2022-08-17T12:09:02.2887107Z d9c9fd17b60c: Download complete 2022-08-17T12:09:02.5159743Z 337a233ac028: Verifying Checksum 2022-08-17T12:09:02.5160211Z 337a233ac028: Download complete 2022-08-17T12:09:02.5362581Z aa1eabed864f: Download complete 2022-08-17T12:09:02.6087108Z 07f2aa52159a: Verifying Checksum 2022-08-17T12:09:02.6087547Z 07f2aa52159a: Download complete 2022-08-17T12:09:02.6408245Z c483592d5074: Verifying Checksum 2022-08-17T12:09:02.6408707Z c483592d5074: Download complete 2022-08-17T12:09:02.7422137Z 1e128c6311be: Verifying Checksum 2022-08-17T12:09:02.7422487Z 1e128c6311be: Download complete 2022-08-17T12:09:02.8216497Z 0a364ea25255: Download complete 2022-08-17T12:09:02.8919983Z 9b8f9a96d4e8: Download complete 2022-08-17T12:09:02.9742562Z 708682444671: Verifying Checksum 2022-08-17T12:09:02.9743176Z 708682444671: Download complete 2022-08-17T12:09:03.0462750Z 589913fd2607: Verifying Checksum 2022-08-17T12:09:03.0463222Z 589913fd2607: Download complete 2022-08-17T12:09:03.8304474Z 7b3f9f48221a: Verifying Checksum 2022-08-17T12:09:03.8304946Z 7b3f9f48221a: Download complete 2022-08-17T12:09:05.9366936Z 13f262da6a6c: Verifying Checksum 2022-08-17T12:09:05.9367248Z 13f262da6a6c: Download complete 2022-08-17T12:09:06.5821355Z 0cd966808131: Pull complete 2022-08-17T12:09:06.8239194Z d96ffce08e66: Pull complete 2022-08-17T12:09:07.0035426Z 6c71e48d181e: Pull complete 2022-08-17T12:09:07.2373114Z 3e0259f02183: Pull complete 2022-08-17T12:09:07.4717262Z 9dc5c789901b: Pull complete 2022-08-17T12:09:07.7224943Z 4ec60a8f4b0d: Pull complete 2022-08-17T12:09:07.9587627Z efb8d44a1237: Pull complete 2022-08-17T12:09:08.1831354Z 0846fd5f50b1: Pull complete 2022-08-17T12:09:10.0350481Z 2f02b9c38fdd: Pull complete 2022-08-17T12:09:10.1746269Z 5e6332363e8c: Pull complete 2022-08-17T12:09:10.3831825Z 7864323978cb: Pull complete 2022-08-17T12:09:13.5609824Z 4ce5b8109bcb: Verifying Checksum 2022-08-17T12:09:13.5610140Z 4ce5b8109bcb: Download complete 2022-08-17T12:09:37.8076028Z 4ce5b8109bcb: Pull complete 2022-08-17T12:09:37.8781855Z 9a5798f00c55: Pull complete 2022-08-17T12:09:37.9586307Z b477457c93ae: Pull complete 2022-08-17T12:09:38.7214419Z b8835462091e: Pull complete 2022-08-17T12:09:38.7984357Z 5c88d6b28819: Pull complete 2022-08-17T12:09:38.9208976Z 8b38f067771b: Pull complete 2022-08-17T12:09:38.9984365Z b5dbbce4b04e: Pull complete 2022-08-17T12:09:39.0956945Z 00a3d832f66e: Pull complete 2022-08-17T12:09:39.1867602Z 682b682d9e93: Pull complete 2022-08-17T12:09:39.6465520Z f62d6a29ee25: Pull complete 2022-08-17T12:09:39.7300488Z 9f2aa5b7f4ee: Pull complete 2022-08-17T12:09:39.8294829Z c1cc323d4a98: Pull complete 2022-08-17T12:09:41.7632808Z d323eef1e7e1: Pull complete 2022-08-17T12:09:41.9725116Z ade3e8c67ae6: Pull complete 2022-08-17T12:09:42.1859306Z a492728f4507: Pull complete 2022-08-17T12:09:42.3890251Z fd9ccd86a681: Pull complete 2022-08-17T12:09:42.5520140Z d932c4ca853a: Pull complete 2022-08-17T12:09:42.7706079Z 15c8440a96aa: Pull complete 2022-08-17T12:09:47.3764291Z 337a233ac028: Pull complete 2022-08-17T12:09:47.4699928Z 97be5b091dd9: Pull complete 2022-08-17T12:09:47.5666757Z 2ada12616b22: Pull complete 2022-08-17T12:09:47.6435564Z e69a2cbb6fd4: Pull complete 2022-08-17T12:09:47.7142862Z 0044ee4ac606: Pull complete 2022-08-17T12:09:47.8032654Z e601349e520b: Pull complete 2022-08-17T12:09:47.9070174Z 591a21ed2ab8: Pull complete 2022-08-17T12:09:47.9908901Z e4521e0c3854: Pull complete 2022-08-17T12:09:48.0907833Z f568ee9f41e3: Pull complete 2022-08-17T12:09:48.1985329Z 19df37b4c74c: Pull complete 2022-08-17T12:09:48.2864536Z c363a482236a: Pull complete 2022-08-17T12:09:48.3871272Z dcf43507e80e: Pull complete 2022-08-17T12:09:48.4754004Z cdeff5e8900e: Pull complete 2022-08-17T12:09:49.6238426Z 432ead746a51: Pull complete 2022-08-17T12:09:49.7312868Z 7887f902330e: Pull complete 2022-08-17T12:09:49.8386886Z 0107d101d703: Pull complete 2022-08-17T12:09:49.9230630Z 424513ba624b: Pull complete 2022-08-17T12:09:50.0069707Z d9c9fd17b60c: Pull complete 2022-08-17T12:09:50.3129711Z aa1eabed864f: Pull complete 2022-08-17T12:09:50.4308219Z c483592d5074: Pull complete 2022-08-17T12:09:50.5228698Z 07f2aa52159a: Pull complete 2022-08-17T12:09:54.5385577Z 13f262da6a6c: Pull complete 2022-08-17T12:09:54.7770708Z 1e128c6311be: Pull complete 2022-08-17T12:09:54.9702501Z 0a364ea25255: Pull complete 2022-08-17T12:09:55.1830164Z 9b8f9a96d4e8: Pull complete 2022-08-17T12:09:55.4170862Z 708682444671: Pull complete 2022-08-17T12:09:55.6450270Z 589913fd2607: Pull complete 2022-08-17T12:09:57.1860484Z 7b3f9f48221a: Pull complete 2022-08-17T12:09:57.1955826Z Digest: sha256:594372be36f848ede98fd4597227797d5fa59cc4886edb7b024f352b98e83e00 2022-08-17T12:09:57.1980672Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:09:57.2003399Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:09:57.2056539Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-08-17T12:09:57.2056820Z python3 -m pip install psutil==5.9.1 2022-08-17T12:09:57.2057071Z python3 -m pip install pynvml==11.4.1 2022-08-17T12:09:57.2057344Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-08-17T12:09:57.2057626Z echo "::set-output name=monitor-script-pid::${!}" 2022-08-17T12:09:57.2069327Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:09:57.2069563Z env: 2022-08-17T12:09:57.2069740Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:09:57.2069941Z ##[endgroup] 2022-08-17T12:09:57.7291709Z Defaulting to user installation because normal site-packages is not writeable 2022-08-17T12:09:57.9937296Z Collecting psutil==5.9.1 2022-08-17T12:09:58.0105880Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2022-08-17T12:09:58.0595101Z Installing collected packages: psutil 2022-08-17T12:09:58.1900793Z Successfully installed psutil-5.9.1 2022-08-17T12:09:58.5594957Z Defaulting to user installation because normal site-packages is not writeable 2022-08-17T12:09:58.6127008Z Collecting pynvml==11.4.1 2022-08-17T12:09:58.6250279Z Downloading pynvml-11.4.1-py3-none-any.whl (46 kB) 2022-08-17T12:09:58.6601590Z Installing collected packages: pynvml 2022-08-17T12:09:58.7061048Z Successfully installed pynvml-11.4.1 2022-08-17T12:09:58.7608896Z Prepare all required actions 2022-08-17T12:09:58.7609166Z Getting action download info 2022-08-17T12:09:58.9332790Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5) 2022-08-17T12:09:59.1044923Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869) 2022-08-17T12:09:59.2178603Z ##[group]Run ./.github/actions/download-build-artifacts 2022-08-17T12:09:59.2178986Z with: 2022-08-17T12:09:59.2179322Z name: linux-focal-py3.7-gcc7 2022-08-17T12:09:59.2179642Z env: 2022-08-17T12:09:59.2179910Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:09:59.2180197Z ##[endgroup] 2022-08-17T12:09:59.2211532Z ##[group]Run seemethere/download-artifact-s3@v4 2022-08-17T12:09:59.2211762Z with: 2022-08-17T12:09:59.2211966Z name: linux-focal-py3.7-gcc7 2022-08-17T12:09:59.2212171Z s3-bucket: gha-artifacts 2022-08-17T12:09:59.2212375Z region: us-east-1 2022-08-17T12:09:59.2212554Z env: 2022-08-17T12:09:59.2212757Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:09:59.2212938Z ##[endgroup] 2022-08-17T12:09:59.6238550Z Found 1 objects with prefix pytorch/pytorch/2875102080/linux-focal-py3.7-gcc7/ 2022-08-17T12:09:59.6239017Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-08-17T12:10:02.8409497Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-08-17T12:10:02.8409835Z 2022-08-17T12:10:02.8410645Z Artifact download has finished successfully 2022-08-17T12:10:02.8504663Z ##[group]Run unzip -o artifacts.zip 2022-08-17T12:10:02.8504911Z unzip -o artifacts.zip 2022-08-17T12:10:02.8516432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:10:02.8516661Z env: 2022-08-17T12:10:02.8516846Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:10:02.8517028Z ##[endgroup] 2022-08-17T12:10:02.8566659Z Archive: artifacts.zip 2022-08-17T12:10:02.8567871Z creating: dist/ 2022-08-17T12:10:03.6311427Z inflating: dist/torch-1.13.0a0+gitce6a3c6-cp37-cp37m-linux_x86_64.whl 2022-08-17T12:10:03.6311757Z creating: build/custom_test_artifacts/ 2022-08-17T12:10:03.6312092Z creating: build/custom_test_artifacts/custom-op-build/ 2022-08-17T12:10:03.6312539Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-08-17T12:10:03.6314552Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-08-17T12:10:03.6314959Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-08-17T12:10:03.6315398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-08-17T12:10:03.6315845Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-08-17T12:10:03.6316284Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-08-17T12:10:03.6318029Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-08-17T12:10:03.6319374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-08-17T12:10:03.6319817Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-08-17T12:10:03.6320431Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-08-17T12:10:03.6322030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-08-17T12:10:03.6323330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-08-17T12:10:03.6324729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-08-17T12:10:03.6325219Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-08-17T12:10:03.6326921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-08-17T12:10:03.6327874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-08-17T12:10:03.6328507Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-08-17T12:10:03.6329102Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-08-17T12:10:03.6329720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-08-17T12:10:03.6330280Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-08-17T12:10:03.6330887Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-08-17T12:10:03.6331701Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-08-17T12:10:03.6332362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-08-17T12:10:03.6332938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-08-17T12:10:03.6333390Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-08-17T12:10:03.6333849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-08-17T12:10:03.6334315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-08-17T12:10:03.6334779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-08-17T12:10:03.6335394Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-08-17T12:10:03.6349255Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-08-17T12:10:03.6441869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-08-17T12:10:03.6442465Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-08-17T12:10:03.6443106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-08-17T12:10:03.6443689Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-08-17T12:10:03.6444352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-08-17T12:10:03.6446334Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-08-17T12:10:03.6446821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-08-17T12:10:03.6447301Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-08-17T12:10:03.6447783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-08-17T12:10:03.6448249Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-08-17T12:10:03.6448729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-08-17T12:10:03.6461653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-08-17T12:10:03.6527262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-08-17T12:10:03.6528156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-08-17T12:10:03.6528993Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-08-17T12:10:03.6529768Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-08-17T12:10:03.6530588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-08-17T12:10:03.6531025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-08-17T12:10:03.6531447Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-08-17T12:10:03.6531951Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-08-17T12:10:03.6532585Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-08-17T12:10:03.6607102Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-08-17T12:10:03.6656978Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-08-17T12:10:03.6657394Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-08-17T12:10:03.6657766Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-08-17T12:10:03.6660072Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-08-17T12:10:03.6660631Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-08-17T12:10:03.6661135Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-08-17T12:10:03.6661611Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-08-17T12:10:03.6662046Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-08-17T12:10:03.6663432Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-08-17T12:10:03.6665183Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-08-17T12:10:03.6665701Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-08-17T12:10:03.6666142Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-08-17T12:10:03.6667752Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-08-17T12:10:03.6669588Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-08-17T12:10:03.6671180Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-08-17T12:10:03.6671706Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-08-17T12:10:03.6673999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-08-17T12:10:03.6674595Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-08-17T12:10:03.6675130Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-08-17T12:10:03.6675947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-08-17T12:10:03.6676853Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-08-17T12:10:03.6677735Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-08-17T12:10:03.6678634Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-08-17T12:10:03.6679579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-08-17T12:10:03.6680566Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-08-17T12:10:03.6681421Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-08-17T12:10:03.6682342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-08-17T12:10:03.6683238Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-08-17T12:10:03.6684091Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-08-17T12:10:03.6685020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-08-17T12:10:03.6686057Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-08-17T12:10:03.6701490Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-08-17T12:10:03.6756923Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-08-17T12:10:03.6757914Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-08-17T12:10:03.6758811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-08-17T12:10:03.6759575Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-08-17T12:10:03.6760300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-08-17T12:10:03.6761087Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-08-17T12:10:03.6762499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-08-17T12:10:03.6763485Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-08-17T12:10:03.6764201Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-08-17T12:10:03.6806788Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-08-17T12:10:03.6807183Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-08-17T12:10:03.6807558Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-08-17T12:10:03.6810097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-08-17T12:10:03.6810885Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-08-17T12:10:03.6811581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-08-17T12:10:03.6812027Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-08-17T12:10:03.6812483Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-08-17T12:10:03.6813353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-08-17T12:10:03.6815129Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-08-17T12:10:03.6815966Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-08-17T12:10:03.6816514Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-08-17T12:10:03.6818209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-08-17T12:10:03.6819466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-08-17T12:10:03.6821278Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-08-17T12:10:03.6822093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-08-17T12:10:03.6823222Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-08-17T12:10:03.6824335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-08-17T12:10:03.6825148Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-08-17T12:10:03.6825919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-08-17T12:10:03.6826734Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-08-17T12:10:03.6827637Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-08-17T12:10:03.6828545Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-08-17T12:10:03.6829479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-08-17T12:10:03.6830424Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-08-17T12:10:03.6830953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-08-17T12:10:03.6831447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-08-17T12:10:03.6831919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-08-17T12:10:03.6832501Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-08-17T12:10:03.6833004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-08-17T12:10:03.6833564Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-08-17T12:10:03.6834437Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-08-17T12:10:03.6956133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-08-17T12:10:03.6957032Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-08-17T12:10:03.6957951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-08-17T12:10:03.6958920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-08-17T12:10:03.6959860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-08-17T12:10:03.6960724Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-08-17T12:10:03.6961235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-08-17T12:10:03.6961723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-08-17T12:10:03.6962236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-08-17T12:10:03.6962743Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-08-17T12:10:03.6963237Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-08-17T12:10:03.6977710Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-08-17T12:10:03.7029266Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-08-17T12:10:03.7030246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-08-17T12:10:03.7031334Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-08-17T12:10:03.7032108Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-08-17T12:10:03.7032824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-08-17T12:10:03.7033260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-08-17T12:10:03.7033931Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-08-17T12:10:03.7034909Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-08-17T12:10:03.7035637Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-08-17T12:10:03.7135427Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-08-17T12:10:03.7172580Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-08-17T12:10:03.7173063Z creating: build/lib/ 2022-08-17T12:10:03.7173466Z inflating: build/lib/libclog.a 2022-08-17T12:10:03.7228845Z inflating: build/lib/libgtest.a 2022-08-17T12:10:03.7238222Z inflating: build/lib/libpthreadpool.a 2022-08-17T12:10:03.7316232Z inflating: build/lib/libbenchmark.a 2022-08-17T12:10:03.7403003Z inflating: build/lib/libprotobuf-lite.a 2022-08-17T12:10:03.7410374Z inflating: build/lib/libittnotify.a 2022-08-17T12:10:03.7436604Z inflating: build/lib/libtensorpipe_uv.a 2022-08-17T12:10:03.7532140Z inflating: build/lib/libgloo.a 2022-08-17T12:10:03.7970341Z inflating: build/lib/libprotobuf.a 2022-08-17T12:10:03.8033137Z inflating: build/lib/libasmjit.a 2022-08-17T12:10:03.8049056Z inflating: build/lib/libfmt.a 2022-08-17T12:10:03.8049674Z inflating: build/lib/libfoxi_loader.a 2022-08-17T12:10:03.8107308Z inflating: build/lib/libc10.so 2022-08-17T12:10:03.8108404Z inflating: build/lib/libtorch_global_deps.so 2022-08-17T12:10:03.8116302Z inflating: build/lib/libcpuinfo.a 2022-08-17T12:10:03.8124117Z inflating: build/lib/libcpuinfo_internals.a 2022-08-17T12:10:03.8137190Z inflating: build/lib/libqnnpack.a 2022-08-17T12:10:03.8139152Z inflating: build/lib/libnnpack_reference_layers.a 2022-08-17T12:10:03.8607660Z inflating: build/lib/libprotoc.a 2022-08-17T12:10:03.8622892Z inflating: build/lib/libgmock.a 2022-08-17T12:10:03.8642552Z inflating: build/lib/libpytorch_qnnpack.a 2022-08-17T12:10:03.8643170Z inflating: build/lib/libgtest_main.a 2022-08-17T12:10:03.8643696Z inflating: build/lib/libbenchmark_main.a 2022-08-17T12:10:04.5321114Z inflating: build/lib/libdnnl.a 2022-08-17T12:10:04.5339130Z inflating: build/lib/libnnpack.a 2022-08-17T12:10:04.5874635Z inflating: build/lib/libtensorpipe.a 2022-08-17T12:10:04.7117244Z inflating: build/lib/libfbgemm.a 2022-08-17T12:10:04.8040406Z inflating: build/lib/libdnnl_graph.a 2022-08-17T12:10:04.8040943Z inflating: build/lib/libgmock_main.a 2022-08-17T12:10:04.8242670Z inflating: build/lib/libkineto.a 2022-08-17T12:10:04.8279804Z inflating: build/lib/libcaffe2_protos.a 2022-08-17T12:10:04.8318993Z inflating: build/lib/libonnx_proto.a 2022-08-17T12:10:04.8435469Z inflating: build/lib/libXNNPACK.a 2022-08-17T12:10:04.9019787Z inflating: build/lib/libonnx.a 2022-08-17T12:10:06.7603021Z inflating: build/lib/libtorch_cpu.so 2022-08-17T12:10:06.7604131Z inflating: build/lib/libtorch.so 2022-08-17T12:10:06.7624103Z inflating: build/lib/libjitbackend_test.so 2022-08-17T12:10:06.7668381Z inflating: build/lib/libtorchbind_test.so 2022-08-17T12:10:06.7693911Z inflating: build/lib/libbackend_with_compiler.so 2022-08-17T12:10:06.7697800Z inflating: build/lib/libshm.so 2022-08-17T12:10:06.9028586Z inflating: build/lib/libtorch_python.so 2022-08-17T12:10:06.9061189Z inflating: build/lib/libnnapi_backend.so 2022-08-17T12:10:06.9061435Z creating: build/bin/ 2022-08-17T12:10:06.9105047Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-08-17T12:10:06.9157078Z inflating: build/bin/c10_DeviceGuard_test 2022-08-17T12:10:06.9216967Z inflating: build/bin/c10_Device_test 2022-08-17T12:10:06.9279120Z inflating: build/bin/c10_DispatchKeySet_test 2022-08-17T12:10:06.9321554Z inflating: build/bin/c10_StreamGuard_test 2022-08-17T12:10:06.9366141Z inflating: build/bin/c10_SymInt_test 2022-08-17T12:10:06.9415526Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-08-17T12:10:06.9464916Z inflating: build/bin/c10_InlineStreamGuard_test 2022-08-17T12:10:06.9516881Z inflating: build/bin/c10_SizesAndStrides_test 2022-08-17T12:10:06.9558908Z inflating: build/bin/c10_Array_test 2022-08-17T12:10:06.9606018Z inflating: build/bin/c10_Bitset_test 2022-08-17T12:10:06.9651568Z inflating: build/bin/c10_C++17_test 2022-08-17T12:10:06.9694177Z inflating: build/bin/c10_ConstexprCrc_test 2022-08-17T12:10:06.9737468Z inflating: build/bin/c10_DeadlockDetection_test 2022-08-17T12:10:06.9781440Z inflating: build/bin/c10_Half_test 2022-08-17T12:10:06.9831541Z inflating: build/bin/c10_LeftRight_test 2022-08-17T12:10:06.9887178Z inflating: build/bin/c10_Metaprogramming_test 2022-08-17T12:10:06.9931447Z inflating: build/bin/c10_Synchronized_test 2022-08-17T12:10:07.0059411Z inflating: build/bin/c10_SmallVectorTest 2022-08-17T12:10:07.0109612Z inflating: build/bin/c10_ThreadLocal_test 2022-08-17T12:10:07.0154449Z inflating: build/bin/c10_TypeList_test 2022-08-17T12:10:07.0196965Z inflating: build/bin/c10_TypeTraits_test 2022-08-17T12:10:07.0243651Z inflating: build/bin/c10_TypeIndex_test 2022-08-17T12:10:07.0289344Z inflating: build/bin/c10_accumulate_test 2022-08-17T12:10:07.0338764Z inflating: build/bin/c10_bfloat16_test 2022-08-17T12:10:07.0386436Z inflating: build/bin/c10_complex_math_test 2022-08-17T12:10:07.0435772Z inflating: build/bin/c10_complex_test 2022-08-17T12:10:07.0532627Z inflating: build/bin/c10_either_test 2022-08-17T12:10:07.0578646Z inflating: build/bin/c10_exception_test 2022-08-17T12:10:07.0622851Z inflating: build/bin/c10_flags_test 2022-08-17T12:10:07.0667620Z inflating: build/bin/c10_irange_test 2022-08-17T12:10:07.0816971Z inflating: build/bin/c10_intrusive_ptr_test 2022-08-17T12:10:07.0868009Z inflating: build/bin/c10_logging_test 2022-08-17T12:10:07.0934256Z inflating: build/bin/c10_optional_test 2022-08-17T12:10:07.0982778Z inflating: build/bin/c10_registry_test 2022-08-17T12:10:07.1038412Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-08-17T12:10:07.1090566Z inflating: build/bin/c10_string_view_test 2022-08-17T12:10:07.1135801Z inflating: build/bin/c10_tempfile_test 2022-08-17T12:10:07.1186006Z inflating: build/bin/c10_typeid_test 2022-08-17T12:10:07.1236030Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-08-17T12:10:07.1669800Z inflating: build/bin/protoc-3.13.0.0 2022-08-17T12:10:07.2102746Z inflating: build/bin/protoc 2022-08-17T12:10:07.2393516Z inflating: build/bin/vec_test_all_types_AVX2 2022-08-17T12:10:07.2654201Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-08-17T12:10:07.2701411Z inflating: build/bin/HashStoreTest 2022-08-17T12:10:07.2754916Z inflating: build/bin/TCPStoreTest 2022-08-17T12:10:07.2802380Z inflating: build/bin/FileStoreTest 2022-08-17T12:10:07.2805050Z inflating: build/bin/example_allreduce 2022-08-17T12:10:07.2862582Z inflating: build/bin/ProcessGroupGlooTest 2022-08-17T12:10:07.2913554Z inflating: build/bin/scalar_test 2022-08-17T12:10:07.2960025Z inflating: build/bin/Dimname_test 2022-08-17T12:10:07.3012813Z inflating: build/bin/apply_utils_test 2022-08-17T12:10:07.3064989Z inflating: build/bin/atest 2022-08-17T12:10:07.3118472Z inflating: build/bin/basic 2022-08-17T12:10:07.3183357Z inflating: build/bin/Dict_test 2022-08-17T12:10:07.3234473Z inflating: build/bin/NamedTensor_test 2022-08-17T12:10:07.3283947Z inflating: build/bin/half_test 2022-08-17T12:10:07.3328968Z inflating: build/bin/wrapdim_test 2022-08-17T12:10:07.3376647Z inflating: build/bin/broadcast_test 2022-08-17T12:10:07.3420493Z inflating: build/bin/dlconvertor_test 2022-08-17T12:10:07.3470403Z inflating: build/bin/scalar_tensor_test 2022-08-17T12:10:07.3520339Z inflating: build/bin/native_test 2022-08-17T12:10:07.3566508Z inflating: build/bin/undefined_tensor_test 2022-08-17T12:10:07.3567877Z inflating: build/bin/verify_api_visibility 2022-08-17T12:10:07.3570027Z inflating: build/bin/thread_init_test 2022-08-17T12:10:07.3619540Z inflating: build/bin/test_parallel 2022-08-17T12:10:07.3670405Z inflating: build/bin/quantized_test 2022-08-17T12:10:07.3715487Z inflating: build/bin/weakref_test 2022-08-17T12:10:07.3759699Z inflating: build/bin/operators_test 2022-08-17T12:10:07.3810152Z inflating: build/bin/extension_backend_test 2022-08-17T12:10:07.3854257Z inflating: build/bin/lazy_tensor_test 2022-08-17T12:10:07.3923318Z inflating: build/bin/tensor_iterator_test 2022-08-17T12:10:07.3970263Z inflating: build/bin/math_kernel_test 2022-08-17T12:10:07.4016808Z inflating: build/bin/memory_overlapping_test 2022-08-17T12:10:07.4063485Z inflating: build/bin/cpu_profiling_allocator_test 2022-08-17T12:10:07.4116184Z inflating: build/bin/cpu_generator_test 2022-08-17T12:10:07.4160477Z inflating: build/bin/variant_test 2022-08-17T12:10:07.4207193Z inflating: build/bin/mobile_memory_cleanup 2022-08-17T12:10:07.4265554Z inflating: build/bin/pow_test 2022-08-17T12:10:07.4310001Z inflating: build/bin/reduce_ops_test 2022-08-17T12:10:07.4357068Z inflating: build/bin/memory_format_test 2022-08-17T12:10:07.4402461Z inflating: build/bin/reportMemoryUsage_test 2022-08-17T12:10:07.4487176Z inflating: build/bin/ivalue_test 2022-08-17T12:10:07.4548193Z inflating: build/bin/vmap_test 2022-08-17T12:10:07.4626627Z inflating: build/bin/cpu_rng_test 2022-08-17T12:10:07.4680711Z inflating: build/bin/type_test 2022-08-17T12:10:07.4725437Z inflating: build/bin/dispatch_key_set_test 2022-08-17T12:10:07.4771466Z inflating: build/bin/stride_properties_test 2022-08-17T12:10:07.4869020Z inflating: build/bin/List_test 2022-08-17T12:10:07.4922449Z inflating: build/bin/IListRef_test 2022-08-17T12:10:07.5029135Z inflating: build/bin/kernel_function_legacy_test 2022-08-17T12:10:07.5086864Z inflating: build/bin/KernelFunction_test 2022-08-17T12:10:07.5172735Z inflating: build/bin/kernel_function_test 2022-08-17T12:10:07.5284721Z inflating: build/bin/kernel_lambda_legacy_test 2022-08-17T12:10:07.5375958Z inflating: build/bin/kernel_lambda_test 2022-08-17T12:10:07.5429664Z inflating: build/bin/kernel_stackbased_test 2022-08-17T12:10:07.5515131Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-08-17T12:10:07.5560039Z inflating: build/bin/CppSignature_test 2022-08-17T12:10:07.5602959Z inflating: build/bin/op_allowlist_test 2022-08-17T12:10:07.5653827Z inflating: build/bin/backend_fallback_test 2022-08-17T12:10:07.5701097Z inflating: build/bin/inline_container_test 2022-08-17T12:10:07.5951790Z inflating: build/bin/op_registration_test 2022-08-17T12:10:07.5966489Z inflating: build/bin/tutorial_tensorexpr 2022-08-17T12:10:07.6014560Z inflating: build/bin/test_dist_autograd 2022-08-17T12:10:07.6076828Z inflating: build/bin/test_cpp_rpc 2022-08-17T12:10:07.6079494Z inflating: build/bin/parallel_benchmark 2022-08-17T12:10:07.6141601Z inflating: build/bin/test_mobile_nnc 2022-08-17T12:10:07.6859568Z inflating: build/bin/test_tensorexpr 2022-08-17T12:10:07.6869091Z inflating: build/bin/aot_model_compiler_test 2022-08-17T12:10:07.7182792Z inflating: build/bin/test_lazy 2022-08-17T12:10:07.8238279Z inflating: build/bin/test_api 2022-08-17T12:10:07.8243189Z inflating: build/bin/torch_shm_manager 2022-08-17T12:10:07.8749990Z inflating: build/bin/test_jit 2022-08-17T12:10:07.8751170Z inflating: .pytorch-test-times.json 2022-08-17T12:10:07.8774291Z ##[group]Run df -H 2022-08-17T12:10:07.8774485Z df -H 2022-08-17T12:10:07.8785486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T12:10:07.8785711Z env: 2022-08-17T12:10:07.8785894Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:10:07.8786078Z ##[endgroup] 2022-08-17T12:10:07.8818376Z Filesystem Size Used Avail Use% Mounted on 2022-08-17T12:10:07.8818828Z devtmpfs 8.1G 0 8.1G 0% /dev 2022-08-17T12:10:07.8819202Z tmpfs 8.2G 0 8.2G 0% /dev/shm 2022-08-17T12:10:07.8819532Z tmpfs 8.2G 463k 8.2G 1% /run 2022-08-17T12:10:07.8819763Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2022-08-17T12:10:07.8820015Z /dev/nvme0n1p1 162G 16G 147G 10% / 2022-08-17T12:10:07.8820227Z tmpfs 1.7G 0 1.7G 0% /run/user/0 2022-08-17T12:10:07.8893318Z ##[group]Run .github/scripts/parse_ref.py 2022-08-17T12:10:07.8893574Z .github/scripts/parse_ref.py 2022-08-17T12:10:07.8904027Z shell: /usr/bin/bash -e {0} 2022-08-17T12:10:07.8904209Z env: 2022-08-17T12:10:07.8904392Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:10:07.8904586Z ##[endgroup] 2022-08-17T12:10:07.9130762Z ##[group]Run set -x 2022-08-17T12:10:07.9131027Z set -x 2022-08-17T12:10:07.9131199Z  2022-08-17T12:10:07.9131408Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-08-17T12:10:07.9131662Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-08-17T12:10:07.9131935Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-08-17T12:10:07.9132185Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-08-17T12:10:07.9132393Z else 2022-08-17T12:10:07.9132590Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-08-17T12:10:07.9132799Z fi 2022-08-17T12:10:07.9132966Z  2022-08-17T12:10:07.9133197Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-08-17T12:10:07.9133553Z  2022-08-17T12:10:07.9133778Z # sanitize the input commit message and PR body here: 2022-08-17T12:10:07.9133990Z # 2022-08-17T12:10:07.9134287Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-08-17T12:10:07.9134682Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-08-17T12:10:07.9134998Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-08-17T12:10:07.9135235Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-08-17T12:10:07.9135429Z  2022-08-17T12:10:07.9135685Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-08-17T12:10:07.9135974Z # wreak havoc internally 2022-08-17T12:10:07.9136220Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-08-17T12:10:07.9136474Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-08-17T12:10:07.9136658Z  2022-08-17T12:10:07.9136893Z # detached container should get cleaned up by teardown_ec2_linux 2022-08-17T12:10:07.9137199Z # TODO: Stop building test binaries as part of the build phase 2022-08-17T12:10:07.9137470Z # Used for GPU_FLAG since that doesn't play nice 2022-08-17T12:10:07.9137719Z # shellcheck disable=SC2086,SC2090 2022-08-17T12:10:07.9137947Z container_name=$(docker run \ 2022-08-17T12:10:07.9138155Z  ${GPU_FLAG:-} \ 2022-08-17T12:10:07.9138348Z  -e BUILD_ENVIRONMENT \ 2022-08-17T12:10:07.9138552Z  -e PR_NUMBER \ 2022-08-17T12:10:07.9138751Z  -e GITHUB_ACTIONS \ 2022-08-17T12:10:07.9138934Z  -e BASE_SHA \ 2022-08-17T12:10:07.9139122Z  -e BRANCH \ 2022-08-17T12:10:07.9139302Z  -e SHA1 \ 2022-08-17T12:10:07.9139571Z  -e AWS_DEFAULT_REGION \ 2022-08-17T12:10:07.9139776Z  -e IN_WHEEL_TEST \ 2022-08-17T12:10:07.9139972Z  -e SHARD_NUMBER \ 2022-08-17T12:10:07.9140156Z  -e TEST_CONFIG \ 2022-08-17T12:10:07.9140356Z  -e NUM_TEST_SHARDS \ 2022-08-17T12:10:07.9140550Z  -e PR_BODY \ 2022-08-17T12:10:07.9140739Z  -e COMMIT_MESSAGES \ 2022-08-17T12:10:07.9140959Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-08-17T12:10:07.9141197Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-08-17T12:10:07.9141403Z  -e PR_LABELS \ 2022-08-17T12:10:07.9141620Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-08-17T12:10:07.9141838Z  -e SCCACHE_BUCKET \ 2022-08-17T12:10:07.9142033Z  -e XLA_CUDA \ 2022-08-17T12:10:07.9142236Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-08-17T12:10:07.9142496Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-08-17T12:10:07.9142741Z  --ulimit stack=10485760:83886080 \ 2022-08-17T12:10:07.9142967Z  --security-opt seccomp=unconfined \ 2022-08-17T12:10:07.9143196Z  --cap-add=SYS_PTRACE \ 2022-08-17T12:10:07.9143397Z  --ipc=host \ 2022-08-17T12:10:07.9143586Z  --shm-size="${SHM_SIZE}" \ 2022-08-17T12:10:07.9143780Z  --tty \ 2022-08-17T12:10:07.9144003Z  --detach \ 2022-08-17T12:10:07.9144196Z  --name="${container_name}" \ 2022-08-17T12:10:07.9144403Z  --user jenkins \ 2022-08-17T12:10:07.9144650Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-08-17T12:10:07.9144912Z  -w /var/lib/jenkins/workspace \ 2022-08-17T12:10:07.9145112Z  "${DOCKER_IMAGE}" 2022-08-17T12:10:07.9145293Z ) 2022-08-17T12:10:07.9145548Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}" 2022-08-17T12:10:07.9156250Z shell: /usr/bin/bash -e {0} 2022-08-17T12:10:07.9156438Z env: 2022-08-17T12:10:07.9156623Z GIT_DEFAULT_BRANCH: master 2022-08-17T12:10:07.9156854Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-08-17T12:10:07.9157077Z PR_NUMBER: 82657 2022-08-17T12:10:07.9157265Z BRANCH: pull/82657 2022-08-17T12:10:07.9157479Z SHA1: ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T12:10:07.9157735Z BASE_SHA: 343b5f86512f75f8f3bd4b90749c0459743b9e72 2022-08-17T12:10:07.9157960Z PYTORCH_RETRY_TEST_CASES: 1 2022-08-17T12:10:07.9158164Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-08-17T12:10:07.9158378Z TEST_CONFIG: distributed 2022-08-17T12:10:07.9158566Z SHARD_NUMBER: 1 2022-08-17T12:10:07.9158733Z NUM_TEST_SHARDS: 1 2022-08-17T12:10:07.9159301Z PR_BODY: ### Description This PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser. Partitioning of the graph is removed because it's handled by the `execute(..., executor="nvfuser")` function. ### Testing Existing tests in `test/test_fx_backends.py`. 2022-08-17T12:10:07.9159900Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-08-17T12:10:07.9160134Z SHM_SIZE: 1g 2022-08-17T12:10:07.9160479Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:10:07.9160802Z XLA_CUDA: 2022-08-17T12:10:07.9161078Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-08-17T12:10:07.9161349Z ##[endgroup] 2022-08-17T12:10:07.9185025Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-08-17T12:10:07.9185474Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]] 2022-08-17T12:10:07.9185707Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-08-17T12:10:07.9188075Z ++ git cherry -v origin/master 2022-08-17T12:10:07.9290313Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx 2022-08-17T12:10:07.9290915Z + 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table 2022-08-17T12:10:07.9291571Z + e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser 2022-08-17T12:10:07.9291994Z + a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec 2022-08-17T12:10:07.9292355Z + fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests 2022-08-17T12:10:07.9292911Z + 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path 2022-08-17T12:10:07.9293283Z + d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py 2022-08-17T12:10:07.9293741Z + ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test' 2022-08-17T12:10:07.9299874Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test' 2022-08-17T12:10:07.9303344Z + PR_BODY='### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because it'\''s handled by the `execute(..., executor="nvfuser")` function.### TestingExisting tests in `test/test_fx_backends.py`.' 2022-08-17T12:10:07.9308397Z + export 'COMMIT_MESSAGES=+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test' 2022-08-17T12:10:07.9310685Z + COMMIT_MESSAGES='+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test' 2022-08-17T12:10:07.9312220Z + export 'PR_BODY=### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`.' 2022-08-17T12:10:07.9313469Z + PR_BODY='### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`.' 2022-08-17T12:10:07.9318941Z +++ nproc --ignore=2 2022-08-17T12:10:07.9445276Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2875102080 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T12:10:18.8953361Z + container_name=b92e618f4d3253d0f3d67ca620cf26031d749b51c0f266aa01cec17e92ef61f4 2022-08-17T12:10:18.8954091Z + docker exec -t b92e618f4d3253d0f3d67ca620cf26031d749b51c0f266aa01cec17e92ef61f4 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-08-17T12:10:19.3200964Z Processing ./dist/torch-1.13.0a0+gitce6a3c6-cp37-cp37m-linux_x86_64.whl 2022-08-17T12:10:19.3941976Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+gitce6a3c6) (4.3.0) 2022-08-17T12:10:19.8590236Z Installing collected packages: torch 2022-08-17T12:10:26.5155709Z Successfully installed torch-1.13.0a0+gitce6a3c6 2022-08-17T12:10:26.5945734Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-08-17T12:10:26.6108645Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-08-17T12:10:26.6109200Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-08-17T12:10:26.6109559Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-08-17T12:10:26.6110030Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-08-17T12:10:26.6110362Z + BUILD_DIR=build 2022-08-17T12:10:26.6110555Z + BUILD_RENAMED_DIR=build_renamed 2022-08-17T12:10:26.6110821Z + BUILD_BIN_DIR=build/bin 2022-08-17T12:10:26.6111109Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]] 2022-08-17T12:10:26.6111354Z ++ realpath build/custom_test_artifacts 2022-08-17T12:10:26.6117053Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2022-08-17T12:10:26.6119711Z ++ dirname .jenkins/pytorch/test.sh 2022-08-17T12:10:26.6125173Z + source .jenkins/pytorch/common.sh 2022-08-17T12:10:26.6128215Z +++ dirname .jenkins/pytorch/common.sh 2022-08-17T12:10:26.6133131Z ++ source .jenkins/pytorch/common_utils.sh 2022-08-17T12:10:26.6134789Z +++ declare -f -t trap_add 2022-08-17T12:10:26.6138573Z ++ set -ex 2022-08-17T12:10:26.6138915Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-08-17T12:10:26.6139179Z ++ BUILD_TEST_LIBTORCH=0 2022-08-17T12:10:26.6140004Z ++ [[ distributed == *xla* ]] 2022-08-17T12:10:26.6140458Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-08-17T12:10:26.6140815Z ++ [[ linux-focal-py3.7-gcc7 == *linux-bionic* ]] 2022-08-17T12:10:26.6141153Z ++ [[ linux-focal-py3.7-gcc7 == *linux-focal* ]] 2022-08-17T12:10:26.6141420Z ++ which conda 2022-08-17T12:10:26.6147932Z /opt/conda/bin/conda 2022-08-17T12:10:26.6148789Z ++ conda install -q -y cmake 2022-08-17T12:10:28.6166951Z Collecting package metadata (current_repodata.json): ...working... done 2022-08-17T12:10:29.0450897Z Solving environment: ...working... done 2022-08-17T12:10:29.1303385Z 2022-08-17T12:10:29.1303555Z ## Package Plan ## 2022-08-17T12:10:29.1303778Z 2022-08-17T12:10:29.1303884Z environment location: /opt/conda 2022-08-17T12:10:29.1304028Z 2022-08-17T12:10:29.1304116Z added / updated specs: 2022-08-17T12:10:29.1304424Z - cmake 2022-08-17T12:10:29.1304534Z 2022-08-17T12:10:29.1304539Z 2022-08-17T12:10:29.1304862Z The following packages will be downloaded: 2022-08-17T12:10:29.1305048Z 2022-08-17T12:10:29.1305199Z package | build 2022-08-17T12:10:29.1305496Z ---------------------------|----------------- 2022-08-17T12:10:29.1305827Z bzip2-1.0.8 | h7b6447c_0 78 KB 2022-08-17T12:10:29.1306131Z c-ares-1.18.1 | h7f8727e_0 114 KB 2022-08-17T12:10:29.1306439Z cmake-3.22.1 | h1fce559_0 7.3 MB 2022-08-17T12:10:29.1306740Z expat-2.4.4 | h295c915_0 169 KB 2022-08-17T12:10:29.1307024Z krb5-1.19.2 | hac12032_0 1.2 MB 2022-08-17T12:10:29.1307324Z libcurl-7.84.0 | h91b91d3_0 337 KB 2022-08-17T12:10:29.1307633Z libedit-3.1.20210910 | h7f8727e_0 166 KB 2022-08-17T12:10:29.1307918Z libev-4.33 | h7f8727e_1 111 KB 2022-08-17T12:10:29.1308230Z libnghttp2-1.46.0 | hce63b2e_0 680 KB 2022-08-17T12:10:29.1308544Z libssh2-1.10.0 | h8f2d780_0 274 KB 2022-08-17T12:10:29.1308841Z libuv-1.40.0 | h7b6447c_0 736 KB 2022-08-17T12:10:29.1309122Z lz4-c-1.9.3 | h295c915_1 185 KB 2022-08-17T12:10:29.1309494Z rhash-1.4.1 | h3c74f83_1 203 KB 2022-08-17T12:10:29.1309789Z zstd-1.5.2 | ha4553b6_0 488 KB 2022-08-17T12:10:29.1310093Z ------------------------------------------------------------ 2022-08-17T12:10:29.1310348Z Total: 12.0 MB 2022-08-17T12:10:29.1310479Z 2022-08-17T12:10:29.1310597Z The following NEW packages will be INSTALLED: 2022-08-17T12:10:29.1310747Z 2022-08-17T12:10:29.1324151Z bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 2022-08-17T12:10:29.1324578Z c-ares pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 2022-08-17T12:10:29.1324996Z cmake pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 2022-08-17T12:10:29.1325372Z expat pkgs/main/linux-64::expat-2.4.4-h295c915_0 2022-08-17T12:10:29.1325736Z krb5 pkgs/main/linux-64::krb5-1.19.2-hac12032_0 2022-08-17T12:10:29.1326321Z libcurl pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0 2022-08-17T12:10:29.1326712Z libedit pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 2022-08-17T12:10:29.1327089Z libev pkgs/main/linux-64::libev-4.33-h7f8727e_1 2022-08-17T12:10:29.1327468Z libnghttp2 pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 2022-08-17T12:10:29.1327862Z libssh2 pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 2022-08-17T12:10:29.1328236Z libuv pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 2022-08-17T12:10:29.1328600Z lz4-c pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 2022-08-17T12:10:29.1328946Z rhash pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 2022-08-17T12:10:29.1329310Z zstd pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 2022-08-17T12:10:29.1329465Z 2022-08-17T12:10:29.1329469Z 2022-08-17T12:10:29.9353141Z Preparing transaction: ...working... done 2022-08-17T12:10:30.3457523Z Verifying transaction: ...working... done 2022-08-17T12:10:30.6421361Z Executing transaction: ...working... done 2022-08-17T12:10:30.8768678Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]] 2022-08-17T12:10:30.8769241Z + echo 'Environment variables' 2022-08-17T12:10:30.8769470Z Environment variables 2022-08-17T12:10:30.8769650Z + env 2022-08-17T12:10:30.8786717Z INSTALLED_DB=yes 2022-08-17T12:10:30.8787625Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-08-17T12:10:30.8788133Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7 2022-08-17T12:10:30.8788513Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-08-17T12:10:30.8788804Z HOSTNAME=b92e618f4d32 2022-08-17T12:10:30.8789409Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_b8575557-bfa0-476a-a046-b6f2baad1428 2022-08-17T12:10:30.8790099Z GITHUB_ACTION=__self 2022-08-17T12:10:30.8790403Z GITHUB_RUN_NUMBER=40757 2022-08-17T12:10:30.8790703Z TEST_CONFIG=distributed 2022-08-17T12:10:30.8791096Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2022-08-17T12:10:30.8791480Z GITHUB_TRIGGERING_ACTOR=IvanYashchuk 2022-08-17T12:10:30.8804095Z GITHUB_REF_TYPE=branch 2022-08-17T12:10:30.8804522Z TORCH_CUDA_ARCH_LIST=Maxwell 2022-08-17T12:10:30.8804970Z BASE_SHA=343b5f86512f75f8f3bd4b90749c0459743b9e72 2022-08-17T12:10:30.8805210Z XLA_CUDA= 2022-08-17T12:10:30.8806304Z *** 2022-08-17T12:10:30.8806497Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2022-08-17T12:10:30.8806706Z GITHUB_ACTIONS=true 2022-08-17T12:10:30.8806923Z SHA1=ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T12:10:30.8807185Z GITHUB_SHA=e9efe573711d9f8141e152403ee143f5875669e7 2022-08-17T12:10:30.8807406Z UCC_HOME=/usr 2022-08-17T12:10:30.8807589Z GITHUB_REF=refs/pull/82657/merge 2022-08-17T12:10:30.8807900Z SHARD_NUMBER=1 2022-08-17T12:10:30.8808242Z GITHUB_REF_PROTECTED=false 2022-08-17T12:10:30.8808587Z HOME=/var/lib/jenkins 2022-08-17T12:10:30.8808846Z GITHUB_API_URL=https://api.github.com 2022-08-17T12:10:30.8809064Z INSTALLED_THRIFT= 2022-08-17T12:10:30.8809246Z UCX_COMMIT=v1.13.x 2022-08-17T12:10:30.8809454Z NUM_TEST_SHARDS=1 2022-08-17T12:10:30.8809768Z UCX_HOME=/usr 2022-08-17T12:10:30.8809967Z PYTORCH_RETRY_TEST_CASES=1 2022-08-17T12:10:30.8810446Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_b8575557-bfa0-476a-a046-b6f2baad1428 2022-08-17T12:10:30.8810910Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2022-08-17T12:10:30.8811198Z GITHUB_EVENT_NAME=pull_request 2022-08-17T12:10:30.8811390Z GITHUB_RUN_ID=2875102080 2022-08-17T12:10:30.8811829Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_b8575557-bfa0-476a-a046-b6f2baad1428 2022-08-17T12:10:30.8847081Z GITHUB_ACTOR=IvanYashchuk 2022-08-17T12:10:30.8847501Z PR_NUMBER=82657 2022-08-17T12:10:30.8847804Z DESIRED_CUDA= 2022-08-17T12:10:30.8848118Z GITHUB_RUN_ATTEMPT=1 2022-08-17T12:10:30.8848525Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-08-17T12:10:30.8848916Z TERM=xterm 2022-08-17T12:10:30.8849207Z INSTALLED_VISION=yes 2022-08-17T12:10:30.8849495Z BRANCH=pull/82657 2022-08-17T12:10:30.8849802Z OPENSSL_ROOT_DIR=/opt/openssl 2022-08-17T12:10:30.8850135Z CUDA_PATH=/usr/local/cuda 2022-08-17T12:10:30.8850833Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2022-08-17T12:10:30.8851358Z GITHUB_SERVER_URL=https://github.com 2022-08-17T12:10:30.8851805Z UCC_COMMIT=a7bda274b10f8adf5bb729f01da064f4e735fb23 2022-08-17T12:10:30.8852178Z INSTALLED_ANDROID= 2022-08-17T12:10:30.8852476Z SHLVL=1 2022-08-17T12:10:30.8852749Z MAX_JOBS=6 2022-08-17T12:10:30.8855383Z COMMIT_MESSAGES=+ 52429d4ec9d162496e677edd78a883288d617a90 Add a helper for make_fx+ 80daf93686965f0383b5fa106ed8782b6905d891 Map torch.ops.aten calls to refs using decomposition_table+ e03d99a2da7e26168d8878f4e77d86f9c1af572a Use TorchRefsNvfuserCapabilityMode for fx.passes.backends.nvfuser+ a2242f67da754bd2714e056764d0cea51f8cdaed Fix output of PartitionedInterpreter to use _out_spec+ fcca9240398d1cf986a5606a61c50f7903d4ee96 Remove incorrect asserts from tests+ 209563d78b9998025726c92d4fa419151454f79e Add a link to the discussion about torch.ops.aten->refs path+ d3f1f74e60a1d82de78a545b7a7a177e3de6c8a2 Revert changes to torch/fx/passes/backends/nvfuser.py+ ce6a3c605df99d1df57c0dda75c06d748e54ed2a Add aten->refs->prims under TorchRefsMode test 2022-08-17T12:10:30.8857128Z GITHUB_REF_NAME=82657/merge 2022-08-17T12:10:30.8857815Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-08-17T12:10:30.8858250Z GITHUB_JOB=test 2022-08-17T12:10:30.8858604Z GITHUB_REPOSITORY=pytorch/pytorch 2022-08-17T12:10:30.8858957Z GITHUB_RETENTION_DAYS=90 2022-08-17T12:10:30.8859254Z OPENSSL_DIR=/opt/openssl 2022-08-17T12:10:30.8859794Z GITHUB_ACTION_REPOSITORY= 2022-08-17T12:10:30.8860356Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-08-17T12:10:30.8860880Z GITHUB_BASE_REF=master 2022-08-17T12:10:30.8861161Z CI=true 2022-08-17T12:10:30.8861495Z GITHUB_REPOSITORY_OWNER=pytorch 2022-08-17T12:10:30.8861874Z INSTALLED_PROTOBUF=yes 2022-08-17T12:10:30.8862339Z GITHUB_HEAD_REF=fx-passes-nvfuser 2022-08-17T12:10:30.8862698Z GITHUB_ACTION_REF= 2022-08-17T12:10:30.8863223Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-08-17T12:10:30.8863644Z GITHUB_WORKFLOW=pull 2022-08-17T12:10:30.8864011Z DEBIAN_FRONTEND=noninteractive 2022-08-17T12:10:30.8865091Z PR_BODY=### DescriptionThis PR replaces `DecompositionInterpreter` with a context manager which decomposes a function into prims only if the decomposition is executable by nvFuser.Partitioning of the graph is removed because its handled by the `execute(..., executor=nvfuser)` function.### TestingExisting tests in `test/test_fx_backends.py`. 2022-08-17T12:10:30.8865993Z _=/usr/bin/env 2022-08-17T12:10:30.8866356Z + echo 'Testing pytorch' 2022-08-17T12:10:30.8866553Z Testing pytorch 2022-08-17T12:10:30.8866874Z + export LANG=C.UTF-8 2022-08-17T12:10:30.8867173Z + LANG=C.UTF-8 2022-08-17T12:10:30.8867438Z + PR_NUMBER=82657 2022-08-17T12:10:30.8867927Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-08-17T12:10:30.8868157Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-08-17T12:10:30.8868556Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-08-17T12:10:30.8868781Z + [[ distributed == \s\l\o\w ]] 2022-08-17T12:10:30.8869067Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]] 2022-08-17T12:10:30.8869369Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-08-17T12:10:30.8869652Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-08-17T12:10:30.8870054Z + [[ linux-focal-py3.7-gcc7 == *cuda11* ]] 2022-08-17T12:10:30.8870763Z + [[ distributed == *crossref* ]] 2022-08-17T12:10:30.8871359Z + [[ distributed == *dynamo* ]] 2022-08-17T12:10:30.8871789Z + [[ -n 82657 ]] 2022-08-17T12:10:30.8871992Z + [[ -z '' ]] 2022-08-17T12:10:30.8872295Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-08-17T12:10:30.8872550Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1 2022-08-17T12:10:30.8872847Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-08-17T12:10:30.8873135Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]] 2022-08-17T12:10:30.8873397Z + pip_install --user ninja 2022-08-17T12:10:30.8873675Z + pip install --progress-bar off --user ninja 2022-08-17T12:10:31.3365618Z Collecting ninja 2022-08-17T12:10:31.3533387Z Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-08-17T12:10:31.7784529Z Installing collected packages: ninja 2022-08-17T12:10:31.7875033Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-08-17T12:10:31.7875594Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-08-17T12:10:31.7927660Z Successfully installed ninja-1.10.2.3 2022-08-17T12:10:31.8668389Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-08-17T12:10:31.8669142Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-08-17T12:10:31.8669839Z + [[ linux-focal-py3.7-gcc7 == *asan* ]] 2022-08-17T12:10:31.8670084Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-08-17T12:10:31.8670331Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-08-17T12:10:31.8673803Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]] 2022-08-17T12:10:31.8687343Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-08-17T12:10:31.8687655Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]] 2022-08-17T12:10:31.8690443Z + cd test 2022-08-17T12:10:31.8690916Z + python -c 'import torch; print(torch.__config__.show())' 2022-08-17T12:10:32.8240980Z PyTorch built with: 2022-08-17T12:10:32.8241629Z - GCC 7.5 2022-08-17T12:10:32.8242045Z - C++ Version: 201402 2022-08-17T12:10:32.8242643Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-08-17T12:10:32.8243106Z - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-08-17T12:10:32.8243430Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-08-17T12:10:32.8243720Z - LAPACK is enabled (usually provided by MKL) 2022-08-17T12:10:32.8243959Z - NNPACK is enabled 2022-08-17T12:10:32.8244193Z - CPU capability usage: AVX2 2022-08-17T12:10:32.8246991Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2022-08-17T12:10:32.8248857Z 2022-08-17T12:10:32.9979349Z + cd test 2022-08-17T12:10:32.9979936Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-08-17T12:10:33.9276481Z ATen/Parallel: 2022-08-17T12:10:33.9276886Z at::get_num_threads() : 4 2022-08-17T12:10:33.9277136Z at::get_num_interop_threads() : 4 2022-08-17T12:10:33.9277370Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-08-17T12:10:33.9277572Z omp_get_max_threads() : 4 2022-08-17T12:10:33.9278115Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-08-17T12:10:33.9278423Z mkl_get_max_threads() : 4 2022-08-17T12:10:33.9278765Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) 2022-08-17T12:10:33.9279053Z std::thread::hardware_concurrency() : 8 2022-08-17T12:10:33.9279276Z Environment variables: 2022-08-17T12:10:33.9279471Z OMP_NUM_THREADS : [not set] 2022-08-17T12:10:33.9279679Z MKL_NUM_THREADS : [not set] 2022-08-17T12:10:33.9279891Z ATen parallel backend: OpenMP 2022-08-17T12:10:33.9280025Z 2022-08-17T12:10:34.1129448Z + [[ distributed == *deploy* ]] 2022-08-17T12:10:34.1129850Z + [[ distributed == *backward* ]] 2022-08-17T12:10:34.1130249Z + [[ distributed == *xla* ]] 2022-08-17T12:10:34.1130616Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-08-17T12:10:34.1130979Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-08-17T12:10:34.1131323Z + [[ distributed == distributed ]] 2022-08-17T12:10:34.1131588Z + install_torchdynamo 2022-08-17T12:10:34.1131765Z + local commit 2022-08-17T12:10:34.1132249Z ++ get_pinned_commit torchdynamo 2022-08-17T12:10:34.1132682Z ++ cat .github/ci_commit_pins/torchdynamo.txt 2022-08-17T12:10:34.1160954Z + commit=f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:34.1161585Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:34.1162225Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:34.4793117Z Collecting git+https://github.com/pytorch/torchdynamo.git@f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:34.4800393Z Cloning https://github.com/pytorch/torchdynamo.git (to revision f19410cd8204fa1c30ca72f81142508e128be66f) to /tmp/pip-req-build-auf8m42s 2022-08-17T12:10:34.5008203Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-auf8m42s 2022-08-17T12:10:35.0726957Z Running command git rev-parse -q --verify 'sha^f19410cd8204fa1c30ca72f81142508e128be66f' 2022-08-17T12:10:35.0763666Z Running command git fetch -q https://github.com/pytorch/torchdynamo.git f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:35.2777720Z Running command git checkout -q f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:35.5616192Z Resolved https://github.com/pytorch/torchdynamo.git to commit f19410cd8204fa1c30ca72f81142508e128be66f 2022-08-17T12:10:38.1618276Z Preparing metadata (setup.py) ... [?25l- \ | done 2022-08-17T12:10:38.1697241Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitce6a3c6) 2022-08-17T12:10:38.1701041Z Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.18.5) 2022-08-17T12:10:38.2069021Z Collecting tabulate 2022-08-17T12:10:38.2233467Z Downloading tabulate-0.8.10-py3-none-any.whl (29 kB) 2022-08-17T12:10:38.2279970Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (6.0) 2022-08-17T12:10:38.2283142Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1) 2022-08-17T12:10:38.2304968Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0) 2022-08-17T12:10:38.2329617Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1) 2022-08-17T12:10:38.2459345Z Building wheels for collected packages: torchdynamo 2022-08-17T12:10:42.9887343Z Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - \ | / - \ done 2022-08-17T12:10:42.9968990Z [?25h Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp37-cp37m-linux_x86_64.whl size=2557816 sha256=12d8c545165b57b8ed7df0c203eaf23e581075f46181e99c3975c2e8f02680a1 2022-08-17T12:10:42.9970600Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/39/47/7c/07229a5bdf22c7a30fc298e692ab525681565397eb89db3641 2022-08-17T12:10:42.9998731Z Successfully built torchdynamo 2022-08-17T12:10:43.4015271Z Installing collected packages: tabulate, torchdynamo 2022-08-17T12:10:43.6501835Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0 2022-08-17T12:10:43.7363416Z + test_distributed 2022-08-17T12:10:43.7363871Z + echo 'Testing distributed python tests' 2022-08-17T12:10:43.7364119Z Testing distributed python tests 2022-08-17T12:10:43.7364470Z + python test/run_test.py --distributed-tests --shard 1 1 --verbose 2022-08-17T12:10:45.0730949Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-08-17T12:10:45.1343084Z Ignoring disabled issues: [] 2022-08-17T12:10:45.1466657Z Selected tests: 2022-08-17T12:10:45.1466987Z distributed/_shard/checkpoint/test_checkpoint 2022-08-17T12:10:45.1467472Z distributed/_shard/checkpoint/test_file_system_checkpoint 2022-08-17T12:10:45.1468001Z distributed/_shard/checkpoint/test_file_system_checkpoint_cpu 2022-08-17T12:10:45.1468482Z distributed/_shard/checkpoint/test_utils 2022-08-17T12:10:45.1468825Z distributed/_shard/sharded_optim/test_sharded_optim 2022-08-17T12:10:45.1469106Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2022-08-17T12:10:45.1469371Z distributed/_shard/sharded_tensor/ops/test_chunk 2022-08-17T12:10:45.1469638Z distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-08-17T12:10:45.1469919Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-08-17T12:10:45.1470277Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-08-17T12:10:45.1470870Z distributed/_shard/sharded_tensor/ops/test_init 2022-08-17T12:10:45.1471299Z distributed/_shard/sharded_tensor/ops/test_linear 2022-08-17T12:10:45.1471655Z distributed/_shard/sharded_tensor/ops/test_math_ops 2022-08-17T12:10:45.1471951Z distributed/_shard/sharded_tensor/ops/test_matrix_ops 2022-08-17T12:10:45.1472391Z distributed/_shard/sharded_tensor/ops/test_softmax 2022-08-17T12:10:45.1472711Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-08-17T12:10:45.1472979Z distributed/_shard/sharded_tensor/test_megatron_prototype 2022-08-17T12:10:45.1473260Z distributed/_shard/sharded_tensor/test_sharded_tensor 2022-08-17T12:10:45.1473540Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-08-17T12:10:45.1473819Z distributed/_shard/sharding_plan/test_sharding_plan 2022-08-17T12:10:45.1474075Z distributed/_shard/sharding_spec/test_sharding_spec 2022-08-17T12:10:45.1474333Z distributed/_shard/test_partial_tensor 2022-08-17T12:10:45.1474580Z distributed/_shard/test_replicated_tensor 2022-08-17T12:10:45.1474797Z distributed/_shard/test_sharder 2022-08-17T12:10:45.1475053Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-08-17T12:10:45.1475333Z distributed/algorithms/quantization/test_quantization 2022-08-17T12:10:45.1475669Z distributed/algorithms/test_join 2022-08-17T12:10:45.1475902Z distributed/elastic/events/lib_test 2022-08-17T12:10:45.1476137Z distributed/elastic/metrics/api_test 2022-08-17T12:10:45.1476510Z distributed/elastic/multiprocessing/api_test 2022-08-17T12:10:45.1476742Z distributed/elastic/timer/api_test 2022-08-17T12:10:45.1476991Z distributed/elastic/timer/local_timer_example 2022-08-17T12:10:45.1477243Z distributed/elastic/timer/local_timer_test 2022-08-17T12:10:45.1477479Z distributed/elastic/utils/distributed_test 2022-08-17T12:10:45.1477720Z distributed/elastic/utils/logging_test 2022-08-17T12:10:45.1477953Z distributed/elastic/utils/util_test 2022-08-17T12:10:45.1478185Z distributed/fsdp/test_checkpoint_wrapper 2022-08-17T12:10:45.1478441Z distributed/fsdp/test_distributed_checkpoint 2022-08-17T12:10:45.1478696Z distributed/fsdp/test_flatten_params_wrapper 2022-08-17T12:10:45.1478919Z distributed/fsdp/test_fsdp_apply 2022-08-17T12:10:45.1479147Z distributed/fsdp/test_fsdp_checkpoint 2022-08-17T12:10:45.1479391Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-08-17T12:10:45.1479625Z distributed/fsdp/test_fsdp_comm 2022-08-17T12:10:45.1479838Z distributed/fsdp/test_fsdp_comm_hooks 2022-08-17T12:10:45.1480063Z distributed/fsdp/test_fsdp_core 2022-08-17T12:10:45.1480286Z distributed/fsdp/test_fsdp_exec_order 2022-08-17T12:10:45.1480520Z distributed/fsdp/test_fsdp_freezing_weights 2022-08-17T12:10:45.1480752Z distributed/fsdp/test_fsdp_fx 2022-08-17T12:10:45.1480974Z distributed/fsdp/test_fsdp_grad_acc 2022-08-17T12:10:45.1481202Z distributed/fsdp/test_fsdp_ignored_modules 2022-08-17T12:10:45.1481436Z distributed/fsdp/test_fsdp_input 2022-08-17T12:10:45.1481661Z distributed/fsdp/test_fsdp_memory 2022-08-17T12:10:45.1481869Z distributed/fsdp/test_fsdp_meta 2022-08-17T12:10:45.1482086Z distributed/fsdp/test_fsdp_misc 2022-08-17T12:10:45.1482321Z distributed/fsdp/test_fsdp_mixed_precision 2022-08-17T12:10:45.1482559Z distributed/fsdp/test_fsdp_multiple_forward 2022-08-17T12:10:45.1482815Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-08-17T12:10:45.1483057Z distributed/fsdp/test_fsdp_optim_state 2022-08-17T12:10:45.1483275Z distributed/fsdp/test_fsdp_overlap 2022-08-17T12:10:45.1483521Z distributed/fsdp/test_fsdp_param_exec_order_wrap 2022-08-17T12:10:45.1483767Z distributed/fsdp/test_fsdp_pure_fp16 2022-08-17T12:10:45.1484004Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-08-17T12:10:45.1484246Z distributed/fsdp/test_fsdp_state_dict 2022-08-17T12:10:45.1484491Z distributed/fsdp/test_fsdp_summon_full_params 2022-08-17T12:10:45.1484730Z distributed/fsdp/test_fsdp_traversal 2022-08-17T12:10:45.1484943Z distributed/fsdp/test_fsdp_uneven 2022-08-17T12:10:45.1485210Z distributed/fsdp/test_shard_utils 2022-08-17T12:10:45.1485425Z distributed/fsdp/test_utils 2022-08-17T12:10:45.1485622Z distributed/fsdp/test_wrap 2022-08-17T12:10:45.1486055Z distributed/nn/jit/test_instantiator 2022-08-17T12:10:45.1486443Z distributed/optim/test_zero_redundancy_optimizer 2022-08-17T12:10:45.1486833Z distributed/pipeline/sync/skip/test_api 2022-08-17T12:10:45.1487147Z distributed/pipeline/sync/skip/test_gpipe 2022-08-17T12:10:45.1487516Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-08-17T12:10:45.1488001Z distributed/pipeline/sync/skip/test_leak 2022-08-17T12:10:45.1488457Z distributed/pipeline/sync/skip/test_portal 2022-08-17T12:10:45.1488891Z distributed/pipeline/sync/skip/test_stash_pop 2022-08-17T12:10:45.1489277Z distributed/pipeline/sync/skip/test_tracker 2022-08-17T12:10:45.1489759Z distributed/pipeline/sync/skip/test_verify_skippables 2022-08-17T12:10:45.1490225Z distributed/pipeline/sync/test_balance 2022-08-17T12:10:45.1490648Z distributed/pipeline/sync/test_bugs 2022-08-17T12:10:45.1491073Z distributed/pipeline/sync/test_checkpoint 2022-08-17T12:10:45.1491491Z distributed/pipeline/sync/test_copy 2022-08-17T12:10:45.1491749Z distributed/pipeline/sync/test_deferred_batch_norm 2022-08-17T12:10:45.1491999Z distributed/pipeline/sync/test_dependency 2022-08-17T12:10:45.1492341Z distributed/pipeline/sync/test_inplace 2022-08-17T12:10:45.1492590Z distributed/pipeline/sync/test_microbatch 2022-08-17T12:10:45.1492820Z distributed/pipeline/sync/test_phony 2022-08-17T12:10:45.1493053Z distributed/pipeline/sync/test_pipe 2022-08-17T12:10:45.1493294Z distributed/pipeline/sync/test_pipeline 2022-08-17T12:10:45.1493521Z distributed/pipeline/sync/test_stream 2022-08-17T12:10:45.1493787Z distributed/pipeline/sync/test_transparency 2022-08-17T12:10:45.1494031Z distributed/pipeline/sync/test_worker 2022-08-17T12:10:45.1494277Z distributed/rpc/cuda/test_tensorpipe_agent 2022-08-17T12:10:45.1494500Z distributed/rpc/test_faulty_agent 2022-08-17T12:10:45.1494729Z distributed/rpc/test_share_memory 2022-08-17T12:10:45.1494958Z distributed/rpc/test_tensorpipe_agent 2022-08-17T12:10:45.1495167Z distributed/test_c10d_common 2022-08-17T12:10:45.1495376Z distributed/test_c10d_gloo 2022-08-17T12:10:45.1495583Z distributed/test_c10d_nccl 2022-08-17T12:10:45.1495795Z distributed/test_c10d_object_collectives 2022-08-17T12:10:45.1496019Z distributed/test_c10d_pypg 2022-08-17T12:10:45.1496230Z distributed/test_c10d_spawn_gloo 2022-08-17T12:10:45.1496434Z distributed/test_c10d_spawn_nccl 2022-08-17T12:10:45.1496649Z distributed/test_data_parallel 2022-08-17T12:10:45.1496878Z distributed/test_distributed_spawn 2022-08-17T12:10:45.1497082Z distributed/test_launcher 2022-08-17T12:10:45.1497286Z distributed/test_nccl 2022-08-17T12:10:45.1497491Z distributed/test_pg_wrapper 2022-08-17T12:10:45.1497686Z distributed/test_store 2022-08-17T12:10:45.1552337Z Prioritized test from test file changes. 2022-08-17T12:10:45.1552719Z reordering tests for PR: 2022-08-17T12:10:45.1553294Z prioritized: ['distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag'] 2022-08-17T12:10:45.1561576Z the rest: ['distributed/_shard/checkpoint/test_checkpoint', 'distributed/_shard/checkpoint/test_file_system_checkpoint', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu', 'distributed/_shard/checkpoint/test_utils', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/test_replicated_tensor', 'distributed/_shard/test_sharder', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/algorithms/quantization/test_quantization', 'distributed/algorithms/test_join', 'distributed/elastic/events/lib_test', 'distributed/elastic/metrics/api_test', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/utils/logging_test', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_misc', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_param_exec_order_wrap', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_shard_utils', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_wrap', 'distributed/nn/jit/test_instantiator', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_worker', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/rpc/test_faulty_agent', 'distributed/rpc/test_share_memory', 'distributed/rpc/test_tensorpipe_agent', 'distributed/test_c10d_common', 'distributed/test_c10d_gloo', 'distributed/test_c10d_nccl', 'distributed/test_c10d_object_collectives', 'distributed/test_c10d_pypg', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/test_data_parallel', 'distributed/test_distributed_spawn', 'distributed/test_launcher', 'distributed/test_nccl', 'distributed/test_pg_wrapper', 'distributed/test_store'] 2022-08-17T12:10:45.1566870Z 2022-08-17T12:10:45.1567309Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2022-08-17T12:10:45.1744067Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2022-08-17T12:10:45.2131688Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-08-17 12:10:45.212904] 2022-08-17T12:10:45.2132334Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:10:45.212952] 2022-08-17T12:10:46.3201857Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding 2022-08-17T12:10:46.3213775Z 2022-08-17T12:10:46.3214102Z Running tests... 2022-08-17T12:10:46.3214763Z ---------------------------------------------------------------------- 2022-08-17T12:10:46.3228621Z test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:46.3242707Z test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:46.3243143Z 2022-08-17T12:10:46.3243535Z ---------------------------------------------------------------------- 2022-08-17T12:10:46.3244176Z Ran 2 tests in 0.003s 2022-08-17T12:10:46.3244379Z 2022-08-17T12:10:46.3244505Z OK (skipped=2) 2022-08-17T12:10:46.3244696Z 2022-08-17T12:10:46.3244836Z Generating XML reports... 2022-08-17T12:10:46.3278360Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220817121046.xml 2022-08-17T12:10:46.5409177Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-08-17 12:10:46.540450] 2022-08-17T12:10:46.5409838Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:10:46.540532] 2022-08-17T12:10:47.6472728Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag 2022-08-17T12:10:47.6483174Z 2022-08-17T12:10:47.6483330Z Running tests... 2022-08-17T12:10:47.6483938Z ---------------------------------------------------------------------- 2022-08-17T12:10:47.6488493Z test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:10:47.6492220Z test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:10:47.6492687Z 2022-08-17T12:10:47.6493087Z ---------------------------------------------------------------------- 2022-08-17T12:10:47.6493355Z Ran 2 tests in 0.001s 2022-08-17T12:10:47.6493476Z 2022-08-17T12:10:47.6493538Z OK (skipped=2) 2022-08-17T12:10:47.6493652Z 2022-08-17T12:10:47.6493740Z Generating XML reports... 2022-08-17T12:10:47.6527059Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220817121047.xml 2022-08-17T12:10:47.8664283Z Running distributed/_shard/checkpoint/test_checkpoint ... [2022-08-17 12:10:47.865997] 2022-08-17T12:10:47.8664911Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:10:47.866077] 2022-08-17T12:10:48.9984812Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint 2022-08-17T12:10:49.0004749Z 2022-08-17T12:10:49.0005136Z Running tests... 2022-08-17T12:10:49.0005761Z ---------------------------------------------------------------------- 2022-08-17T12:10:49.0023593Z test_storage_key_mapping (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:10:49.0029888Z test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:49.0035515Z test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:10:49.0044828Z test_load_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:49.2773566Z test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 707 2022-08-17T12:10:49.2803102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 708 2022-08-17T12:10:49.2831198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 709 2022-08-17T12:10:49.2860503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 710 2022-08-17T12:10:50.3591633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:50.3592422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:50.3594691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:50.3595580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:50.3855642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:50.3856418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:50.3857213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:50.3857844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:50.4191476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:50.4192237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:50.4195105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:50.4195717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:50.4292405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:50.4292983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:50.4296160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:50.4296695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:50.5160260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:10:50.5439226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:10:50.5906055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:10:50.5958219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:10:50.8904323Z ok (1.886s) 2022-08-17T12:10:50.8917592Z test_save_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:50.8968862Z test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 751 2022-08-17T12:10:50.9003163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 752 2022-08-17T12:10:50.9032824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 753 2022-08-17T12:10:50.9069275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 754 2022-08-17T12:10:51.9573249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:51.9573928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:51.9574718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:51.9575371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:51.9577116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:51.9577773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:51.9578606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:51.9579248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:51.9804828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:51.9805467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:51.9808686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:51.9809634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:51.9874168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:51.9874695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:51.9878665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:51.9879335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:52.1142969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:10:52.1143739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:10:52.1380064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:10:52.1438910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:10:52.4107205Z ok (1.519s) 2022-08-17T12:10:52.4126942Z test_create_key_handles_collision (__main__.TestStorageKeys) ... ok (0.002s) 2022-08-17T12:10:52.4127482Z 2022-08-17T12:10:52.4128152Z ---------------------------------------------------------------------- 2022-08-17T12:10:52.4128586Z Ran 8 tests in 3.412s 2022-08-17T12:10:52.4128785Z 2022-08-17T12:10:52.4128924Z OK (skipped=5) 2022-08-17T12:10:52.4129109Z 2022-08-17T12:10:52.4129253Z Generating XML reports... 2022-08-17T12:10:52.4167458Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220817121048.xml 2022-08-17T12:10:52.4171165Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestStorageKeys-20220817121048.xml 2022-08-17T12:10:52.4176173Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220817121048.xml 2022-08-17T12:10:52.7366127Z Running distributed/_shard/checkpoint/test_file_system_checkpoint ... [2022-08-17 12:10:52.736163] 2022-08-17T12:10:52.7366797Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:10:52.736248] 2022-08-17T12:10:53.8470944Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint 2022-08-17T12:10:53.8485243Z 2022-08-17T12:10:53.8485573Z Running tests... 2022-08-17T12:10:53.8486387Z ---------------------------------------------------------------------- 2022-08-17T12:10:53.8501036Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:53.8524294Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:10:53.8532369Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:53.8551034Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:10:54.1234600Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.268s) 2022-08-17T12:10:54.1246141Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:10:54.1246700Z 2022-08-17T12:10:54.1247187Z ---------------------------------------------------------------------- 2022-08-17T12:10:54.1247619Z Ran 6 tests in 0.276s 2022-08-17T12:10:54.1247814Z 2022-08-17T12:10:54.1247937Z OK (skipped=5) 2022-08-17T12:10:54.1248103Z 2022-08-17T12:10:54.1248252Z Generating XML reports... 2022-08-17T12:10:54.1273718Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220817121053.xml 2022-08-17T12:10:54.1281258Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220817121053.xml 2022-08-17T12:10:54.1284858Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220817121053.xml 2022-08-17T12:10:54.4067399Z Running distributed/_shard/checkpoint/test_file_system_checkpoint_cpu ... [2022-08-17 12:10:54.406364] 2022-08-17T12:10:54.4068090Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:10:54.406449] 2022-08-17T12:10:55.5223345Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu 2022-08-17T12:10:55.5236800Z 2022-08-17T12:10:55.5236890Z Running tests... 2022-08-17T12:10:55.5237407Z ---------------------------------------------------------------------- 2022-08-17T12:10:55.7953780Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 816 2022-08-17T12:10:55.7981508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 817 2022-08-17T12:10:56.7663417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:56.7663847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:56.7665767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:56.7666196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:56.7689254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:56.7689688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:56.7692941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:56.7693366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:56.9271507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:10:56.9288156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:10:56.9480833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:10:56.9481328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:10:56.9481992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:10:56.9482606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:10:57.3016521Z ok (1.777s) 2022-08-17T12:10:57.3086829Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 844 2022-08-17T12:10:57.3121400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 845 2022-08-17T12:10:58.2721633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:58.2722055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:58.2723105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:58.2723523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:58.2724200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:58.2724557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:58.2727211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:58.2727606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:58.4297762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:10:58.4308108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:10:58.4416941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:10:58.4417364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:10:58.4418588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:10:58.4419184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:10:58.9153153Z ok (1.614s) 2022-08-17T12:10:58.9206433Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 872 2022-08-17T12:10:58.9241797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 873 2022-08-17T12:10:59.9004473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:59.9004942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:59.9006410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:59.9006799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:10:59.9081452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:10:59.9081838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:10:59.9084083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:10:59.9084586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:00.0595108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:00.0668774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:00.0778770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:00.0779518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:00.0780195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:00.0780758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:00.4273537Z ok (1.512s) 2022-08-17T12:11:00.4338681Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 900 2022-08-17T12:11:00.4373965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 901 2022-08-17T12:11:01.4345240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:01.4345666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:01.4347408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:01.4347817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:01.4569596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:01.4570029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:01.4572106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:01.4572466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:01.6073379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:01.6253653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:01.6363289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:01.6363973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:01.6364633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:01.6365197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:02.1410242Z ok (1.714s) 2022-08-17T12:11:02.1564481Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.015s) 2022-08-17T12:11:02.1615934Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 928 2022-08-17T12:11:02.1654180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 929 2022-08-17T12:11:03.2304543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:03.2305025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:03.2307754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:03.2308185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:03.2746134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:03.2746750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:03.2748430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:03.2749242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:03.4222794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:03.4327672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:03.4438016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:03.4438558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:03.4439216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:03.4439781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:11:03.7691279Z ok (1.612s) 2022-08-17T12:11:03.7691511Z 2022-08-17T12:11:03.7692063Z ---------------------------------------------------------------------- 2022-08-17T12:11:03.7692419Z Ran 6 tests in 8.245s 2022-08-17T12:11:03.7692552Z 2022-08-17T12:11:03.7692617Z OK 2022-08-17T12:11:03.7692714Z 2022-08-17T12:11:03.7692800Z Generating XML reports... 2022-08-17T12:11:03.7729596Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220817121055.xml 2022-08-17T12:11:03.7730661Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220817121055.xml 2022-08-17T12:11:03.7735636Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220817121055.xml 2022-08-17T12:11:04.1206035Z Running distributed/_shard/checkpoint/test_utils ... [2022-08-17 12:11:04.120114] 2022-08-17T12:11:04.1206737Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:04.120192] 2022-08-17T12:11:05.2838319Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_utils 2022-08-17T12:11:05.2850928Z 2022-08-17T12:11:05.2851241Z Running tests... 2022-08-17T12:11:05.2851906Z ---------------------------------------------------------------------- 2022-08-17T12:11:05.5686539Z test_flat_data (__main__.TestMedatadaIndex) ... ok (0.283s) 2022-08-17T12:11:05.5695728Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-08-17T12:11:05.5703236Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-08-17T12:11:05.5712060Z test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-08-17T12:11:05.5743477Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s) 2022-08-17T12:11:05.5743892Z 2022-08-17T12:11:05.5744322Z ---------------------------------------------------------------------- 2022-08-17T12:11:05.5744715Z Ran 5 tests in 0.289s 2022-08-17T12:11:05.5744936Z 2022-08-17T12:11:05.5745054Z OK 2022-08-17T12:11:05.5745226Z 2022-08-17T12:11:05.5745388Z Generating XML reports... 2022-08-17T12:11:05.5778436Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20220817121105.xml 2022-08-17T12:11:05.9312887Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-08-17 12:11:05.930884] 2022-08-17T12:11:05.9313582Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:05.930968] 2022-08-17T12:11:07.1298996Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim 2022-08-17T12:11:07.1310799Z 2022-08-17T12:11:07.1311195Z Running tests... 2022-08-17T12:11:07.1311998Z ---------------------------------------------------------------------- 2022-08-17T12:11:07.1322533Z test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:07.1338108Z test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:07.1338459Z 2022-08-17T12:11:07.1338917Z ---------------------------------------------------------------------- 2022-08-17T12:11:07.1339283Z Ran 2 tests in 0.003s 2022-08-17T12:11:07.1339403Z 2022-08-17T12:11:07.1339479Z OK (skipped=2) 2022-08-17T12:11:07.1339579Z 2022-08-17T12:11:07.1339669Z Generating XML reports... 2022-08-17T12:11:07.1373530Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220817121107.xml 2022-08-17T12:11:07.3862235Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-08-17 12:11:07.385771] 2022-08-17T12:11:07.3862966Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:07.385851] 2022-08-17T12:11:08.6160944Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp 2022-08-17T12:11:08.6175458Z 2022-08-17T12:11:08.6175759Z Running tests... 2022-08-17T12:11:08.6176455Z ---------------------------------------------------------------------- 2022-08-17T12:11:08.6185742Z test_torch_allclose (__main__.TestShardedTensorBinaryOps) 2022-08-17T12:11:08.6186235Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:08.6189571Z test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:08.6193598Z test_torch_equal (__main__.TestShardedTensorBinaryOps) 2022-08-17T12:11:08.6194069Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:08.6197859Z test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:08.6198287Z 2022-08-17T12:11:08.6198715Z ---------------------------------------------------------------------- 2022-08-17T12:11:08.6199140Z Ran 4 tests in 0.002s 2022-08-17T12:11:08.6199284Z 2022-08-17T12:11:08.6199362Z OK (skipped=4) 2022-08-17T12:11:08.6199476Z 2022-08-17T12:11:08.6199564Z Generating XML reports... 2022-08-17T12:11:08.6234442Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220817121108.xml 2022-08-17T12:11:08.8450910Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-08-17 12:11:08.844687] 2022-08-17T12:11:08.8451545Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:08.844774] 2022-08-17T12:11:10.0537809Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk 2022-08-17T12:11:10.0551635Z 2022-08-17T12:11:10.0551930Z Running tests... 2022-08-17T12:11:10.0552670Z ---------------------------------------------------------------------- 2022-08-17T12:11:10.0562048Z test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:10.0572273Z test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:10.0572574Z 2022-08-17T12:11:10.0573136Z ---------------------------------------------------------------------- 2022-08-17T12:11:10.0573460Z Ran 2 tests in 0.002s 2022-08-17T12:11:10.0573577Z 2022-08-17T12:11:10.0573830Z OK (skipped=2) 2022-08-17T12:11:10.0573946Z 2022-08-17T12:11:10.0574037Z Generating XML reports... 2022-08-17T12:11:10.0609050Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220817121110.xml 2022-08-17T12:11:10.2958135Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-08-17 12:11:10.295384] 2022-08-17T12:11:10.2958808Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:10.295467] 2022-08-17T12:11:11.4730810Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops 2022-08-17T12:11:11.4744533Z 2022-08-17T12:11:11.4744865Z Running tests... 2022-08-17T12:11:11.4745268Z ---------------------------------------------------------------------- 2022-08-17T12:11:11.4755864Z test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:11.4763420Z test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:11.4772605Z test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:11.4780499Z test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:11.4781262Z 2022-08-17T12:11:11.4781818Z ---------------------------------------------------------------------- 2022-08-17T12:11:11.4782277Z Ran 4 tests in 0.004s 2022-08-17T12:11:11.4782499Z 2022-08-17T12:11:11.4782635Z OK (skipped=4) 2022-08-17T12:11:11.4782837Z 2022-08-17T12:11:11.4782999Z Generating XML reports... 2022-08-17T12:11:11.4819107Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220817121111.xml 2022-08-17T12:11:11.7246931Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-08-17 12:11:11.724262] 2022-08-17T12:11:11.7248033Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:11.724364] 2022-08-17T12:11:12.8565575Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2022-08-17T12:11:12.8576851Z 2022-08-17T12:11:12.8577361Z Running tests... 2022-08-17T12:11:12.8577796Z ---------------------------------------------------------------------- 2022-08-17T12:11:12.8589266Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2022-08-17T12:11:12.8591094Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:12.8601450Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2022-08-17T12:11:12.8602412Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:12.8613195Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2022-08-17T12:11:12.8613877Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:12.8614245Z 2022-08-17T12:11:12.8614670Z ---------------------------------------------------------------------- 2022-08-17T12:11:12.8615084Z Ran 3 tests in 0.004s 2022-08-17T12:11:12.8615284Z 2022-08-17T12:11:12.8615403Z OK (skipped=3) 2022-08-17T12:11:12.8615590Z 2022-08-17T12:11:12.8615723Z Generating XML reports... 2022-08-17T12:11:12.8651317Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220817121112.xml 2022-08-17T12:11:13.1069670Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-08-17 12:11:13.106520] 2022-08-17T12:11:13.1070569Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:13.106620] 2022-08-17T12:11:14.3394268Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear 2022-08-17T12:11:14.3406812Z 2022-08-17T12:11:14.3406920Z Running tests... 2022-08-17T12:11:14.3408054Z ---------------------------------------------------------------------- 2022-08-17T12:11:14.3425583Z test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:14.3453812Z test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-08-17T12:11:14.3465440Z test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:14.3465881Z 2022-08-17T12:11:14.3466350Z ---------------------------------------------------------------------- 2022-08-17T12:11:14.3466794Z Ran 3 tests in 0.006s 2022-08-17T12:11:14.3466995Z 2022-08-17T12:11:14.3467132Z OK (skipped=3) 2022-08-17T12:11:14.3467594Z 2022-08-17T12:11:14.3467687Z Generating XML reports... 2022-08-17T12:11:14.3501665Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220817121114.xml 2022-08-17T12:11:14.5646847Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-08-17 12:11:14.564245] 2022-08-17T12:11:14.5647716Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:14.564328] 2022-08-17T12:11:15.8897984Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-08-17 12:11:15.889318] 2022-08-17T12:11:15.8898870Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:15.889400] 2022-08-17T12:11:16.9907945Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops 2022-08-17T12:11:16.9920319Z 2022-08-17T12:11:16.9920601Z Running tests... 2022-08-17T12:11:16.9921235Z ---------------------------------------------------------------------- 2022-08-17T12:11:16.9928566Z test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9940096Z test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9948905Z test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9953867Z test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:16.9963191Z test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9970454Z test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9981263Z test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:16.9986242Z test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:16.9995415Z test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:17.0004972Z test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:17.0013491Z test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:17.0014183Z 2022-08-17T12:11:17.0014648Z ---------------------------------------------------------------------- 2022-08-17T12:11:17.0015067Z Ran 11 tests in 0.009s 2022-08-17T12:11:17.0015294Z 2022-08-17T12:11:17.0015412Z OK (skipped=11) 2022-08-17T12:11:17.0015601Z 2022-08-17T12:11:17.0015689Z Generating XML reports... 2022-08-17T12:11:17.0057381Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220817121116.xml 2022-08-17T12:11:17.2102346Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-08-17 12:11:17.209874] 2022-08-17T12:11:17.2102992Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:17.209952] 2022-08-17T12:11:18.2958586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv5c0yb_ 2022-08-17T12:11:18.2959736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv5c0yb_/_remote_module_non_scriptable.py 2022-08-17T12:11:18.3073388Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax 2022-08-17T12:11:18.3086588Z 2022-08-17T12:11:18.3086728Z Running tests... 2022-08-17T12:11:18.3087164Z ---------------------------------------------------------------------- 2022-08-17T12:11:18.3091722Z test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:18.3095854Z test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:18.3096147Z 2022-08-17T12:11:18.3096408Z ---------------------------------------------------------------------- 2022-08-17T12:11:18.3096652Z Ran 2 tests in 0.001s 2022-08-17T12:11:18.3096776Z 2022-08-17T12:11:18.3096849Z OK (skipped=2) 2022-08-17T12:11:18.3096960Z 2022-08-17T12:11:18.3097054Z Generating XML reports... 2022-08-17T12:11:18.3121967Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220817121118.xml 2022-08-17T12:11:18.5124496Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-08-17 12:11:18.512062] 2022-08-17T12:11:18.5125377Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:18.512141] 2022-08-17T12:11:19.6174088Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2022-08-17T12:11:19.6184998Z 2022-08-17T12:11:19.6185127Z Running tests... 2022-08-17T12:11:19.6185736Z ---------------------------------------------------------------------- 2022-08-17T12:11:19.6192656Z test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:19.6199288Z test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:19.6206572Z test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:19.6215145Z test_inplace_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:19.6222704Z test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:19.6222967Z 2022-08-17T12:11:19.6223332Z ---------------------------------------------------------------------- 2022-08-17T12:11:19.6223585Z Ran 5 tests in 0.004s 2022-08-17T12:11:19.6223912Z 2022-08-17T12:11:19.6223989Z OK (skipped=5) 2022-08-17T12:11:19.6224101Z 2022-08-17T12:11:19.6224188Z Generating XML reports... 2022-08-17T12:11:19.6257828Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220817121119.xml 2022-08-17T12:11:19.8259206Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-08-17 12:11:19.825566] 2022-08-17T12:11:19.8259861Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:19.825647] 2022-08-17T12:11:20.9364388Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype 2022-08-17T12:11:20.9375405Z 2022-08-17T12:11:20.9375528Z Running tests... 2022-08-17T12:11:20.9375955Z ---------------------------------------------------------------------- 2022-08-17T12:11:20.9395202Z test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:20.9395593Z 2022-08-17T12:11:20.9395869Z ---------------------------------------------------------------------- 2022-08-17T12:11:20.9396129Z Ran 1 test in 0.002s 2022-08-17T12:11:20.9396534Z 2022-08-17T12:11:20.9396613Z OK (skipped=1) 2022-08-17T12:11:20.9396717Z 2022-08-17T12:11:20.9396807Z Generating XML reports... 2022-08-17T12:11:20.9428113Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220817121120.xml 2022-08-17T12:11:21.1589657Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-08-17 12:11:21.158551] 2022-08-17T12:11:21.1590313Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:21.158633] 2022-08-17T12:11:22.2870568Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor 2022-08-17T12:11:22.2904369Z 2022-08-17T12:11:22.2904751Z Running tests... 2022-08-17T12:11:22.2905293Z ---------------------------------------------------------------------- 2022-08-17T12:11:22.2913315Z test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s) 2022-08-17T12:11:22.2920239Z test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2926382Z test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2932912Z test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2943297Z test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2948729Z test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:22.2956184Z test_shard_metadata_init (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2968542Z test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2983831Z test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.2992274Z test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3004337Z test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3010377Z test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3028023Z test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3044546Z test_create_sharded_tensor_like (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3045299Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3054495Z test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3055142Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3063047Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3063395Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3077512Z test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3078210Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3086790Z test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3087404Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3096068Z test_gather_even (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3096780Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3105348Z test_gather_uneven (__main__.TestShardedTensorChunked) 2022-08-17T12:11:22.3106038Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3117648Z test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3125158Z test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3145312Z test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3156885Z test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3171185Z test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3187105Z test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3202617Z test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3216836Z test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3234259Z test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3244871Z test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3255132Z test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3264482Z test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3272379Z test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3279676Z test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3284771Z test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:22.3293304Z test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3304755Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable) 2022-08-17T12:11:22.3305431Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3316370Z test_gather_even (__main__.TestShardedTensorEnumerable) 2022-08-17T12:11:22.3316768Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3328374Z test_gather_uneven (__main__.TestShardedTensorEnumerable) 2022-08-17T12:11:22.3328775Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3348546Z test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3370491Z test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3392400Z test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3411934Z test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3418957Z test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3437789Z test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3475357Z test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3476273Z test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.3499194Z test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3520382Z test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3540056Z test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3558301Z test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3582281Z test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:22.3609952Z test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-08-17T12:11:22.3623693Z test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:22.6262119Z test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1087 2022-08-17T12:11:22.6289510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1088 2022-08-17T12:11:22.6318249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1089 2022-08-17T12:11:22.6348405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1090 2022-08-17T12:11:23.6841024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:23.6842052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:23.6844109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:23.6844750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:23.7109568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:23.7110302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:23.7113779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:23.7114508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:23.7220722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:23.7221252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:23.7224324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:23.7224858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:23.7309900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:23.7310617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:23.7313527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:23.7314158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:23.8435962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:23.8699757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:23.8802035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:11:23.8871789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:23.9284211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:23.9284634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:11:23.9386364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:23.9387417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:11:23.9388414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:11:23.9388983Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:11:23.9488424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:11:23.9489297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:11:24.2391928Z skip: Need at least 4 CUDA devices (1.877s) 2022-08-17T12:11:24.2409463Z test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:24.2416134Z test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:24.2422746Z test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:24.2436386Z test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:24.2445390Z test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:24.2510083Z test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1143 2022-08-17T12:11:24.2543536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1144 2022-08-17T12:11:24.2571919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1145 2022-08-17T12:11:24.2599158Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1146 2022-08-17T12:11:25.3008985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:25.3009625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:25.3012407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:25.3013086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:25.3241742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:25.3242444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:25.3243256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:25.3243877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:25.3244708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:25.3245348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:25.3246420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:25.3247066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:25.3483147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:25.3483798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:25.3487909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:25.3488583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:25.4802732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:25.4896167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:25.5004075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:25.5172425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:11:25.7637723Z skip: Need at least 4 CUDA devices (1.519s) 2022-08-17T12:11:25.7645813Z test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:25.7655046Z test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:25.8017772Z test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.036s) 2022-08-17T12:11:25.8018130Z 2022-08-17T12:11:25.8018613Z ---------------------------------------------------------------------- 2022-08-17T12:11:25.8018996Z Ran 64 tests in 3.511s 2022-08-17T12:11:25.8019209Z 2022-08-17T12:11:25.8019337Z OK (skipped=63) 2022-08-17T12:11:25.8019511Z 2022-08-17T12:11:25.8019653Z Generating XML reports... 2022-08-17T12:11:25.8053219Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220817121122.xml 2022-08-17T12:11:25.8056514Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220817121122.xml 2022-08-17T12:11:25.8061437Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220817121122.xml 2022-08-17T12:11:25.8066088Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220817121122.xml 2022-08-17T12:11:25.8071369Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20220817121122.xml 2022-08-17T12:11:25.8075985Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220817121122.xml 2022-08-17T12:11:25.8080656Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220817121122.xml 2022-08-17T12:11:25.8105375Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220817121122.xml 2022-08-17T12:11:25.8110826Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220817121122.xml 2022-08-17T12:11:25.8137297Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220817121122.xml 2022-08-17T12:11:25.8149941Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220817121122.xml 2022-08-17T12:11:25.8155269Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220817121122.xml 2022-08-17T12:11:26.1262335Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-08-17 12:11:26.125862] 2022-08-17T12:11:26.1263312Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:26.125947] 2022-08-17T12:11:27.2532492Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard 2022-08-17T12:11:27.2543711Z 2022-08-17T12:11:27.2543834Z Running tests... 2022-08-17T12:11:27.2544444Z ---------------------------------------------------------------------- 2022-08-17T12:11:27.2552197Z test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:27.2561880Z test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:27.2562561Z 2022-08-17T12:11:27.2562871Z ---------------------------------------------------------------------- 2022-08-17T12:11:27.2563272Z Ran 2 tests in 0.002s 2022-08-17T12:11:27.2563483Z 2022-08-17T12:11:27.2563636Z OK (skipped=2) 2022-08-17T12:11:27.2563840Z 2022-08-17T12:11:27.2563998Z Generating XML reports... 2022-08-17T12:11:27.2594981Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220817121127.xml 2022-08-17T12:11:27.4759158Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-08-17 12:11:27.475445] 2022-08-17T12:11:27.4760033Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:27.475524] 2022-08-17T12:11:28.5917984Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan 2022-08-17T12:11:28.5929131Z 2022-08-17T12:11:28.5929286Z Running tests... 2022-08-17T12:11:28.5929779Z ---------------------------------------------------------------------- 2022-08-17T12:11:28.5938026Z test_custom_sharding_planner (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:28.5953683Z test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:11:28.5961082Z test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:28.5972598Z test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:28.6008867Z test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.004s) 2022-08-17T12:11:28.6009209Z 2022-08-17T12:11:28.6009482Z ---------------------------------------------------------------------- 2022-08-17T12:11:28.6009732Z Ran 5 tests in 0.008s 2022-08-17T12:11:28.6009853Z 2022-08-17T12:11:28.6009928Z OK (skipped=5) 2022-08-17T12:11:28.6012296Z 2022-08-17T12:11:28.6012489Z Generating XML reports... 2022-08-17T12:11:28.6045674Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20220817121128.xml 2022-08-17T12:11:28.8148594Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-08-17 12:11:28.814454] 2022-08-17T12:11:28.8149451Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:28.814533] 2022-08-17T12:11:29.9065385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mg8x9qg 2022-08-17T12:11:29.9066566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mg8x9qg/_remote_module_non_scriptable.py 2022-08-17T12:11:29.9190704Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-08-17T12:11:29.9204703Z 2022-08-17T12:11:29.9204881Z Running tests... 2022-08-17T12:11:29.9205498Z ---------------------------------------------------------------------- 2022-08-17T12:11:30.1840090Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1218 2022-08-17T12:11:30.1867843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1219 2022-08-17T12:11:30.1896050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1220 2022-08-17T12:11:30.1925219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1221 2022-08-17T12:11:31.2340021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:31.2340735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:31.2341556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:31.2342111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:31.2342965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:31.2343610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:31.2346637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:31.2347320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:31.2414026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:31.2415002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:31.2417810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:31.2418443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:31.2419216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:31.2419814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:31.2422676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:31.2423351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:31.3984735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8dbw5sn 2022-08-17T12:11:31.3985567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8dbw5sn/_remote_module_non_scriptable.py 2022-08-17T12:11:31.3990223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nwuak37 2022-08-17T12:11:31.3993066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nwuak37/_remote_module_non_scriptable.py 2022-08-17T12:11:31.4059432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp286y94k 2022-08-17T12:11:31.4060153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcao8l0g 2022-08-17T12:11:31.4061401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp286y94k/_remote_module_non_scriptable.py 2022-08-17T12:11:31.4062071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcao8l0g/_remote_module_non_scriptable.py 2022-08-17T12:11:31.4097996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:31.4106713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:11:31.4174420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:31.4175129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:31.6968278Z ok (1.776s) 2022-08-17T12:11:31.6977057Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-08-17T12:11:31.6978196Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:31.6985132Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-08-17T12:11:31.6985943Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:31.6999996Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-08-17T12:11:31.7007229Z test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s) 2022-08-17T12:11:31.7044689Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.004s) 2022-08-17T12:11:31.7065704Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s) 2022-08-17T12:11:31.7076465Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-08-17T12:11:31.7088660Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-08-17T12:11:31.7163490Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.008s) 2022-08-17T12:11:31.7163819Z 2022-08-17T12:11:31.7164323Z ---------------------------------------------------------------------- 2022-08-17T12:11:31.7164765Z Ran 10 tests in 1.796s 2022-08-17T12:11:31.7164903Z 2022-08-17T12:11:31.7164986Z OK (skipped=5) 2022-08-17T12:11:31.7165100Z 2022-08-17T12:11:31.7165188Z Generating XML reports... 2022-08-17T12:11:31.7199172Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220817121129.xml 2022-08-17T12:11:31.7206930Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220817121129.xml 2022-08-17T12:11:32.0354368Z Running distributed/_shard/test_partial_tensor ... [2022-08-17 12:11:32.034987] 2022-08-17T12:11:32.0355309Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:32.035068] 2022-08-17T12:11:33.1528660Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor 2022-08-17T12:11:33.1540275Z 2022-08-17T12:11:33.1540424Z Running tests... 2022-08-17T12:11:33.1540864Z ---------------------------------------------------------------------- 2022-08-17T12:11:33.1551301Z test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:33.1559509Z test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:33.1563512Z test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:11:33.1574897Z test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:33.1585645Z test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:33.1586127Z 2022-08-17T12:11:33.1586559Z ---------------------------------------------------------------------- 2022-08-17T12:11:33.1586982Z Ran 5 tests in 0.005s 2022-08-17T12:11:33.1587182Z 2022-08-17T12:11:33.1587302Z OK (skipped=5) 2022-08-17T12:11:33.1587490Z 2022-08-17T12:11:33.1587624Z Generating XML reports... 2022-08-17T12:11:33.1620935Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220817121133.xml 2022-08-17T12:11:33.1625498Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220817121133.xml 2022-08-17T12:11:33.3697093Z Running distributed/_shard/test_replicated_tensor ... [2022-08-17 12:11:33.369281] 2022-08-17T12:11:33.3697732Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:33.369360] 2022-08-17T12:11:34.6865659Z Running distributed/_shard/test_sharder ... [2022-08-17 12:11:34.686062] 2022-08-17T12:11:34.6866294Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:34.686143] 2022-08-17T12:11:35.7863689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie7z9om9 2022-08-17T12:11:35.7864477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie7z9om9/_remote_module_non_scriptable.py 2022-08-17T12:11:36.0136748Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-08-17 12:11:36.013234] 2022-08-17T12:11:36.0137444Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:36.013318] 2022-08-17T12:11:37.1256985Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks 2022-08-17T12:11:37.1267760Z 2022-08-17T12:11:37.1267893Z Running tests... 2022-08-17T12:11:37.1268506Z ---------------------------------------------------------------------- 2022-08-17T12:11:37.1276309Z test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest) 2022-08-17T12:11:37.1277088Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1282484Z test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest) 2022-08-17T12:11:37.1282916Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1289871Z test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest) 2022-08-17T12:11:37.1290694Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1296264Z test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest) 2022-08-17T12:11:37.1297068Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1302297Z test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest) 2022-08-17T12:11:37.1302775Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1314835Z test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:37.1315294Z 2022-08-17T12:11:37.1315903Z ---------------------------------------------------------------------- 2022-08-17T12:11:37.1316171Z Ran 6 tests in 0.005s 2022-08-17T12:11:37.1316292Z 2022-08-17T12:11:37.1316370Z OK (skipped=6) 2022-08-17T12:11:37.1316484Z 2022-08-17T12:11:37.1316560Z Generating XML reports... 2022-08-17T12:11:37.1347682Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220817121137.xml 2022-08-17T12:11:37.3419349Z Running distributed/algorithms/quantization/test_quantization ... [2022-08-17 12:11:37.341508] 2022-08-17T12:11:37.3478259Z MPI not available -- MPI backend tests will be skipped 2022-08-17T12:11:37.3482904Z Running distributed tests for the test backend with env init_method 2022-08-17T12:11:37.3486744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:37.348473] 2022-08-17T12:11:38.4362814Z 2022-08-17T12:11:38.6386900Z Running distributed tests for the test backend with file init_method 2022-08-17T12:11:38.6388424Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:38.638608] 2022-08-17T12:11:39.7323221Z 2022-08-17T12:11:39.9425806Z Running distributed tests for the gloo backend with env init_method 2022-08-17T12:11:39.9427185Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:39.942493] 2022-08-17T12:11:41.0426033Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-08-17T12:11:41.0426769Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0427047Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0427339Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0427616Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0427887Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0428187Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:41.0428696Z 2022-08-17T12:11:41.9872878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:41.9873297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:41.9875068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:41.9875447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:42.1403514Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:42.1413415Z 2022-08-17T12:11:42.1413508Z Running tests... 2022-08-17T12:11:42.1415155Z ---------------------------------------------------------------------- 2022-08-17T12:11:42.4028006Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1344 2022-08-17T12:11:42.4056020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1345 2022-08-17T12:11:42.4083866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1346 2022-08-17T12:11:43.4052962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:43.4053670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:43.4055132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:43.4055723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:43.4109475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:43.4110130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:43.4111875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:43.4112651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:43.4259759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:43.4260442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:43.4262018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:43.4262664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:43.5616595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:43.5645687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:43.5794881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:43.5902476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:43.5955675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:11:43.5956086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:43.5956737Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:43.5957299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:43.6005331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:43.9122548Z ok (1.771s) 2022-08-17T12:11:43.9122825Z 2022-08-17T12:11:43.9123351Z ---------------------------------------------------------------------- 2022-08-17T12:11:43.9123836Z Ran 1 test in 1.771s 2022-08-17T12:11:43.9123961Z 2022-08-17T12:11:43.9124024Z OK 2022-08-17T12:11:43.9124122Z 2022-08-17T12:11:43.9124222Z Generating XML reports... 2022-08-17T12:11:43.9161924Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121142.xml 2022-08-17T12:11:45.1700444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:45.1700838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:45.1703042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:45.1703404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:45.3208055Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:45.3218320Z 2022-08-17T12:11:45.3218743Z Running tests... 2022-08-17T12:11:45.3219395Z ---------------------------------------------------------------------- 2022-08-17T12:11:45.5814687Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1397 2022-08-17T12:11:45.5841507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1398 2022-08-17T12:11:45.5868275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1399 2022-08-17T12:11:46.5730345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:46.5730992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:46.5732777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:46.5733204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:46.5896894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:46.5897380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:46.5900431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:46.5901127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:46.6069861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:46.6070528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:46.6071316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:46.6071920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:46.7278349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:46.7438617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:46.7598825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:46.7907844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:46.8010077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:46.8010511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:11:46.8011169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:46.8011721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:46.8111958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:47.1910643Z ok (1.869s) 2022-08-17T12:11:47.1910864Z 2022-08-17T12:11:47.1911293Z ---------------------------------------------------------------------- 2022-08-17T12:11:47.1911584Z Ran 1 test in 1.869s 2022-08-17T12:11:47.1911694Z 2022-08-17T12:11:47.1911757Z OK 2022-08-17T12:11:47.1911853Z 2022-08-17T12:11:47.1911991Z Generating XML reports... 2022-08-17T12:11:47.1947027Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121145.xml 2022-08-17T12:11:48.4906718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:48.4907146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:48.4908763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:48.4909158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:48.6400906Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:48.6410646Z 2022-08-17T12:11:48.6410993Z Running tests... 2022-08-17T12:11:48.6411440Z ---------------------------------------------------------------------- 2022-08-17T12:11:48.6420429Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:48.6420754Z 2022-08-17T12:11:48.6421112Z ---------------------------------------------------------------------- 2022-08-17T12:11:48.6421575Z Ran 1 test in 0.001s 2022-08-17T12:11:48.6421791Z 2022-08-17T12:11:48.6421909Z OK (skipped=1) 2022-08-17T12:11:48.6422099Z 2022-08-17T12:11:48.6422199Z Generating XML reports... 2022-08-17T12:11:48.6446910Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121148.xml 2022-08-17T12:11:49.8129554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:49.8129988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:49.8131090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:49.8131477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:49.9630228Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:49.9639606Z 2022-08-17T12:11:49.9639698Z Running tests... 2022-08-17T12:11:49.9640175Z ---------------------------------------------------------------------- 2022-08-17T12:11:49.9649746Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:49.9650400Z 2022-08-17T12:11:49.9650987Z ---------------------------------------------------------------------- 2022-08-17T12:11:49.9651443Z Ran 1 test in 0.001s 2022-08-17T12:11:49.9651623Z 2022-08-17T12:11:49.9651758Z OK (skipped=1) 2022-08-17T12:11:49.9651946Z 2022-08-17T12:11:49.9652114Z Generating XML reports... 2022-08-17T12:11:49.9676745Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121149.xml 2022-08-17T12:11:51.1126301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:51.1126688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:51.1128572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:51.1128949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:51.2651526Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:51.2661067Z 2022-08-17T12:11:51.2661200Z Running tests... 2022-08-17T12:11:51.2661776Z ---------------------------------------------------------------------- 2022-08-17T12:11:51.2670717Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:51.2671079Z 2022-08-17T12:11:51.2671510Z ---------------------------------------------------------------------- 2022-08-17T12:11:51.2671992Z Ran 1 test in 0.001s 2022-08-17T12:11:51.2672200Z 2022-08-17T12:11:51.2672360Z OK (skipped=1) 2022-08-17T12:11:51.2672476Z 2022-08-17T12:11:51.2672553Z Generating XML reports... 2022-08-17T12:11:51.2696442Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121151.xml 2022-08-17T12:11:52.4181516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:52.4182046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:52.4183153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:52.4183739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:52.5689909Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:52.5699632Z 2022-08-17T12:11:52.5699738Z Running tests... 2022-08-17T12:11:52.5700826Z ---------------------------------------------------------------------- 2022-08-17T12:11:52.5709289Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:11:52.5709847Z 2022-08-17T12:11:52.5710248Z ---------------------------------------------------------------------- 2022-08-17T12:11:52.5710708Z Ran 1 test in 0.001s 2022-08-17T12:11:52.5710931Z 2022-08-17T12:11:52.5711070Z OK (skipped=1) 2022-08-17T12:11:52.5711256Z 2022-08-17T12:11:52.5711379Z Generating XML reports... 2022-08-17T12:11:52.5736987Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121152.xml 2022-08-17T12:11:53.0151688Z Running distributed tests for the gloo backend with file init_method 2022-08-17T12:11:53.0153291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:11:53.015082] 2022-08-17T12:11:54.1163486Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-08-17T12:11:54.1164328Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1164627Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1164918Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1165193Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1165480Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1165772Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-08-17T12:11:54.1166226Z 2022-08-17T12:11:55.0607083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:55.0607508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:55.0608769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:55.0609145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:55.2117884Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:55.2127324Z 2022-08-17T12:11:55.2127685Z Running tests... 2022-08-17T12:11:55.2128111Z ---------------------------------------------------------------------- 2022-08-17T12:11:55.4762512Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1500 2022-08-17T12:11:55.4789974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1501 2022-08-17T12:11:55.4816960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1502 2022-08-17T12:11:56.4819328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:56.4819922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:56.4820577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:56.4821169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:56.4974733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:56.4975096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:56.4977084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:56.4977459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:56.5017191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:56.5017552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:56.5020756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:56.5021132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:56.6395836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:56.6552902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:56.6590528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:56.6762398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:56.6863100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:11:56.6863728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:56.6864723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:56.6865628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:56.6866412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:56.9857469Z ok (1.773s) 2022-08-17T12:11:56.9857663Z 2022-08-17T12:11:56.9857999Z ---------------------------------------------------------------------- 2022-08-17T12:11:56.9858315Z Ran 1 test in 1.773s 2022-08-17T12:11:56.9858427Z 2022-08-17T12:11:56.9858490Z OK 2022-08-17T12:11:56.9858590Z 2022-08-17T12:11:56.9858691Z Generating XML reports... 2022-08-17T12:11:56.9894272Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121155.xml 2022-08-17T12:11:58.2392915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:58.2406278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:58.2407308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:58.2407909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:58.3923292Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:11:58.3933285Z 2022-08-17T12:11:58.3933837Z Running tests... 2022-08-17T12:11:58.3934512Z ---------------------------------------------------------------------- 2022-08-17T12:11:58.6523546Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1553 2022-08-17T12:11:58.6549895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1554 2022-08-17T12:11:58.6576918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1555 2022-08-17T12:11:59.6487506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:59.6488172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:59.6488963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:59.6489583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:59.6545091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:59.6545771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:59.6548459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:59.6549083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:59.6980239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:11:59.6980895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:11:59.6982329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:11:59.6982836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:11:59.8026585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:11:59.8102581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:11:59.8532630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:11:59.8736788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:11:59.8814531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:11:59.8815066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:11:59.8815734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:59.8816296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:11:59.8838521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T12:12:00.2618135Z ok (1.868s) 2022-08-17T12:12:00.2618350Z 2022-08-17T12:12:00.2618886Z ---------------------------------------------------------------------- 2022-08-17T12:12:00.2619353Z Ran 1 test in 1.868s 2022-08-17T12:12:00.2619477Z 2022-08-17T12:12:00.2619540Z OK 2022-08-17T12:12:00.2619638Z 2022-08-17T12:12:00.2619720Z Generating XML reports... 2022-08-17T12:12:00.2657289Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121158.xml 2022-08-17T12:12:01.5227072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:01.5227576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:01.5228622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:01.5228985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:01.6764198Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:12:01.6773854Z 2022-08-17T12:12:01.6774330Z Running tests... 2022-08-17T12:12:01.6774764Z ---------------------------------------------------------------------- 2022-08-17T12:12:01.6783947Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:12:01.6784400Z 2022-08-17T12:12:01.6785138Z ---------------------------------------------------------------------- 2022-08-17T12:12:01.6785410Z Ran 1 test in 0.001s 2022-08-17T12:12:01.6785530Z 2022-08-17T12:12:01.6785606Z OK (skipped=1) 2022-08-17T12:12:01.6785721Z 2022-08-17T12:12:01.6785797Z Generating XML reports... 2022-08-17T12:12:01.6810772Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121201.xml 2022-08-17T12:12:02.8331283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:02.8331787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:02.8333289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:02.8333755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:02.9835650Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:12:02.9844758Z 2022-08-17T12:12:02.9845192Z Running tests... 2022-08-17T12:12:02.9845539Z ---------------------------------------------------------------------- 2022-08-17T12:12:02.9855032Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:12:02.9855351Z 2022-08-17T12:12:02.9855736Z ---------------------------------------------------------------------- 2022-08-17T12:12:02.9856042Z Ran 1 test in 0.001s 2022-08-17T12:12:02.9856216Z 2022-08-17T12:12:02.9856332Z OK (skipped=1) 2022-08-17T12:12:02.9856450Z 2022-08-17T12:12:02.9856549Z Generating XML reports... 2022-08-17T12:12:02.9882249Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121202.xml 2022-08-17T12:12:04.1389557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:04.1390016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:04.1391423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:04.1391966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:04.2902566Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:12:04.2911636Z 2022-08-17T12:12:04.2911830Z Running tests... 2022-08-17T12:12:04.2912187Z ---------------------------------------------------------------------- 2022-08-17T12:12:04.2922360Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:12:04.2922666Z 2022-08-17T12:12:04.2922951Z ---------------------------------------------------------------------- 2022-08-17T12:12:04.2923215Z Ran 1 test in 0.001s 2022-08-17T12:12:04.2923338Z 2022-08-17T12:12:04.2923425Z OK (skipped=1) 2022-08-17T12:12:04.2923541Z 2022-08-17T12:12:04.2923618Z Generating XML reports... 2022-08-17T12:12:04.2948524Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121204.xml 2022-08-17T12:12:05.4424274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:05.4424662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:05.4426403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:05.4426794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:05.5928225Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-08-17T12:12:05.5937921Z 2022-08-17T12:12:05.5938368Z Running tests... 2022-08-17T12:12:05.5939322Z ---------------------------------------------------------------------- 2022-08-17T12:12:05.5948800Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:12:05.5949133Z 2022-08-17T12:12:05.5949604Z ---------------------------------------------------------------------- 2022-08-17T12:12:05.5950037Z Ran 1 test in 0.001s 2022-08-17T12:12:05.5950165Z 2022-08-17T12:12:05.5950263Z OK (skipped=1) 2022-08-17T12:12:05.5950456Z 2022-08-17T12:12:05.5950637Z Generating XML reports... 2022-08-17T12:12:05.5974072Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121205.xml 2022-08-17T12:12:06.0291132Z Running distributed/algorithms/test_join ... [2022-08-17 12:12:06.028669] 2022-08-17T12:12:06.0291751Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:06.028738] 2022-08-17T12:12:07.1261771Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-08-17T12:12:07.1273106Z 2022-08-17T12:12:07.1273206Z Running tests... 2022-08-17T12:12:07.1273789Z ---------------------------------------------------------------------- 2022-08-17T12:12:07.1283678Z test_join_kwargs (__main__.TestJoin) 2022-08-17T12:12:07.3951268Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1646 2022-08-17T12:12:07.3981681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1647 2022-08-17T12:12:08.3730465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:08.3731175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:08.3732328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:08.3732963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:08.3864643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:08.3865294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:08.3867352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:08.3867980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:08.5286633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:08.5409573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:08.5596930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:08.5597610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:08.5598535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:08.5599419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:08.9022490Z ok (1.775s) 2022-08-17T12:12:08.9029526Z test_multiple_joinable_disable (__main__.TestJoin) 2022-08-17T12:12:08.9076216Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1674 2022-08-17T12:12:08.9114674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1675 2022-08-17T12:12:09.8555086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:09.8555497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:09.8556312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:09.8556673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:09.8557486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:09.8557934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:09.8558713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:09.8559141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:10.0084163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:10.0084855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:10.0294019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:10.0294711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:10.0295532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:10.0296098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:10.3146235Z ok (1.412s) 2022-08-17T12:12:10.3154667Z test_multiple_joinables (__main__.TestJoin) 2022-08-17T12:12:10.3200144Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1702 2022-08-17T12:12:10.3235097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1703 2022-08-17T12:12:11.2687371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:11.2687795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:11.2690169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:11.2690687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:11.3008960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:11.3009387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:11.3011893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:11.3012878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:11.4222846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:11.4568746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:11.4733616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:11.4734285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:11.4735221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:11.4736097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:11.8267257Z ok (1.512s) 2022-08-17T12:12:11.8273412Z test_multiple_joinables_throw (__main__.TestJoin) 2022-08-17T12:12:11.8320188Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1730 2022-08-17T12:12:11.8357970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1731 2022-08-17T12:12:12.7865523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:12.7866157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:12.7867445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:12.7868070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:12.7976465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:12.7977115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:12.7979228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:12.7979844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:12.9404415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:12.9524953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:12.9713333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:12.9714013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:12.9714997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:12.9715845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:13.2386568Z ok (1.412s) 2022-08-17T12:12:13.2394707Z test_single_joinable (__main__.TestJoin) 2022-08-17T12:12:13.2440087Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1758 2022-08-17T12:12:13.2474732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1759 2022-08-17T12:12:14.1948986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:14.1949617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:14.1950402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:14.1950903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:14.1977911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:14.1978440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:14.1981152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:14.1981837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:14.3478175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:14.3514175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:14.3723473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:14.3723963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:14.3724667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:14.3725329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:14.6505683Z ok (1.412s) 2022-08-17T12:12:14.6511969Z test_single_joinable_disable (__main__.TestJoin) 2022-08-17T12:12:14.6558032Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1786 2022-08-17T12:12:14.6592603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1787 2022-08-17T12:12:15.6109880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:15.6110375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:15.6112052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:15.6112663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:15.6420172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:15.6420625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:15.6422260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:15.6422791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:15.7646452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:15.7938632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:15.8056746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:15.8057375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:15.8058017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:15.8058600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:16.0625493Z ok (1.412s) 2022-08-17T12:12:16.0633857Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-08-17T12:12:16.0678590Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1814 2022-08-17T12:12:16.0713884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1815 2022-08-17T12:12:17.0240887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:17.0241537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:17.0243082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:17.0243697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:17.0392913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:17.0393575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:17.0395017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:17.0395642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:17.1735720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:17.1887390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:17.2044311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:17.2044973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:17.2046113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:17.2047152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:17.4746716Z ok (1.412s) 2022-08-17T12:12:17.4752688Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-08-17T12:12:17.4797961Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1842 2022-08-17T12:12:17.4832415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1843 2022-08-17T12:12:18.4237516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:18.4238191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:18.4239536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:18.4240135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:18.4258680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:18.4259069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:18.4261598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:18.4261968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:18.5754092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:18.5765676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:18.5961606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:18.5962059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:18.5962713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:18.5963274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:18.8865105Z ok (1.412s) 2022-08-17T12:12:18.8871091Z test_single_joinable_throw (__main__.TestJoin) 2022-08-17T12:12:18.8919784Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1870 2022-08-17T12:12:18.8956411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1871 2022-08-17T12:12:19.8318398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:19.8318994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:19.8327939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:19.8328343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:19.8632258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:19.8632776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:19.8635172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:19.8635636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:19.9922743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:12:20.0172184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:12:20.0331546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:12:20.0332160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:12:20.0332805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:20.0333364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:12:20.2987764Z ok (1.412s) 2022-08-17T12:12:20.2987922Z 2022-08-17T12:12:20.2988768Z ---------------------------------------------------------------------- 2022-08-17T12:12:20.2989147Z Ran 9 tests in 13.171s 2022-08-17T12:12:20.2989258Z 2022-08-17T12:12:20.2989324Z OK 2022-08-17T12:12:20.2989442Z 2022-08-17T12:12:20.2989540Z Generating XML reports... 2022-08-17T12:12:20.3030033Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220817121207.xml 2022-08-17T12:12:20.6192812Z Running distributed/elastic/events/lib_test ... [2022-08-17 12:12:20.618841] 2022-08-17T12:12:20.6193584Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-08-17 12:12:20.618928] 2022-08-17T12:12:21.2066168Z ============================= test session starts ============================== 2022-08-17T12:12:21.2151051Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:12:21.2151502Z cachedir: .pytest_cache 2022-08-17T12:12:21.2152410Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:12:21.2153105Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:12:21.2153840Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:12:22.2231468Z collecting ...  2022-08-17T12:12:22.2241945Z collecting 3 items  2022-08-17T12:12:22.2242318Z collected 8 items  2022-08-17T12:12:22.2246715Z 2022-08-17T12:12:22.2259895Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-08-17T12:12:22.2270401Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-08-17T12:12:22.2283380Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-08-17T12:12:22.2983907Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-08-17T12:12:22.3010484Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-08-17T12:12:22.3019426Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-08-17T12:12:22.3029457Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-08-17T12:12:22.3043681Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-08-17T12:12:22.3045163Z 2022-08-17T12:12:22.3045542Z ============================== 8 passed in 1.10s =============================== 2022-08-17T12:12:22.5460330Z Running distributed/elastic/metrics/api_test ... [2022-08-17 12:12:22.545665] 2022-08-17T12:12:22.5460912Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:22.545745] 2022-08-17T12:12:23.6299485Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-08-17T12:12:23.6309786Z 2022-08-17T12:12:23.6309895Z Running tests... 2022-08-17T12:12:23.6310358Z ---------------------------------------------------------------------- 2022-08-17T12:12:23.8840690Z test_get_metric_name (__main__.MetricsApiTest) ... ok (0.253s) 2022-08-17T12:12:23.8852693Z test_inheritance (__main__.MetricsApiTest) ... ok (0.001s) 2022-08-17T12:12:23.8869455Z test_profile (__main__.MetricsApiTest) ... ok (0.002s) 2022-08-17T12:12:23.8869871Z 2022-08-17T12:12:23.8870287Z ---------------------------------------------------------------------- 2022-08-17T12:12:23.8870637Z Ran 3 tests in 0.256s 2022-08-17T12:12:23.8870762Z 2022-08-17T12:12:23.8870827Z OK 2022-08-17T12:12:23.8870910Z 2022-08-17T12:12:23.8871049Z Generating XML reports... 2022-08-17T12:12:23.8897421Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220817121223.xml 2022-08-17T12:12:24.1689008Z Running distributed/elastic/multiprocessing/api_test ... [2022-08-17 12:12:24.168483] 2022-08-17T12:12:24.1689644Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:24.168565] 2022-08-17T12:12:25.3546497Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-08-17T12:12:25.3559820Z 2022-08-17T12:12:25.3560059Z Running tests... 2022-08-17T12:12:25.3560563Z ---------------------------------------------------------------------- 2022-08-17T12:12:25.6108284Z test_get_failures (__main__.RunProcResultsTest) ... ok (0.255s) 2022-08-17T12:12:25.6117985Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-08-17T12:12:25.6133642Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s) 2022-08-17T12:12:25.6419030Z test_binary (__main__.StartProcessesListTest) ... hello stdout from 0 2022-08-17T12:12:25.6419327Z hello stderr from 0 2022-08-17T12:12:25.6460370Z hello stdout from 1 2022-08-17T12:12:25.6460649Z hello stderr from 1 2022-08-17T12:12:25.7272754Z ok (0.114s) 2022-08-17T12:12:25.7601871Z test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1 2022-08-17T12:12:25.7602142Z bar stderr from 1 2022-08-17T12:12:25.8409343Z failed (exitcode: 138) local_rank: 0 (pid: 1931) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-08-17T12:12:25.8420596Z ok (0.115s) 2022-08-17T12:12:25.8505773Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.008s) 2022-08-17T12:12:25.8774913Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-08-17T12:12:25.8775604Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-08-17T12:12:25.8776084Z raise RuntimeError(f"raised from {rank}") 2022-08-17T12:12:25.8776765Z RuntimeError: raised from 0 2022-08-17T12:12:25.8810125Z bar from 1 2022-08-17T12:12:25.9617506Z failed (exitcode: 1) local_rank: 0 (pid: 1934) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-08-17T12:12:25.9625577Z ok (0.112s) 2022-08-17T12:12:25.9958641Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-08-17T12:12:26.0767810Z [trainer0]:hello stdout from 0 2022-08-17T12:12:26.0768056Z [trainer1]:world stderr from 1 2022-08-17T12:12:27.0791150Z ok (1.116s) 2022-08-17T12:12:28.1515574Z test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:28.1516204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:28.1516943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:28.1517335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:28.1745280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:28.1745665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:28.1748279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:28.1748674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:28.3030998Z hello stdout from 0 2022-08-17T12:12:28.3031342Z hello stderr from 0 2022-08-17T12:12:28.3277660Z hello stdout from 1 2022-08-17T12:12:28.3278020Z hello stderr from 1 2022-08-17T12:12:28.6305730Z Closing process 1942 via signal SIGTERM 2022-08-17T12:12:28.6330108Z ok (1.554s) 2022-08-17T12:12:29.7656174Z test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:29.7656974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:29.7657981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:29.7678871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:29.7680208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:29.7681067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:29.7682312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:29.7682982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:29.7784797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:29.7785688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:29.7786872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:29.7787470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:29.7788227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:29.7788840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:29.7790812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:29.7791440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:30.4903259Z Closing process 1961 via signal SIGTERM 2022-08-17T12:12:30.4903688Z Closing process 1962 via signal SIGTERM 2022-08-17T12:12:30.4904059Z Closing process 1963 via signal SIGTERM 2022-08-17T12:12:30.5129481Z ok (1.879s) 2022-08-17T12:12:30.5145821Z test_function_raise (__main__.StartProcessesListTest) 2022-08-17T12:12:31.6309999Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:31.6310785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:31.6311277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:31.6311652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:31.6452879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:31.6453225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:31.6456432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:31.6457009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:32.0352892Z failed (exitcode: 1) local_rank: 0 (pid: 2001) of fn: echo2 (start_method: spawn) 2022-08-17T12:12:32.0353316Z Traceback (most recent call last): 2022-08-17T12:12:32.0354143Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-08-17T12:12:32.0354721Z self._pc.join(-1) 2022-08-17T12:12:32.0355381Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-08-17T12:12:32.0355752Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-08-17T12:12:32.0356116Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-08-17T12:12:32.0356406Z 2022-08-17T12:12:32.0356685Z -- Process 0 terminated with the following error: 2022-08-17T12:12:32.0356925Z Traceback (most recent call last): 2022-08-17T12:12:32.0357348Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-08-17T12:12:32.0358030Z fn(i, *args) 2022-08-17T12:12:32.0358537Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-08-17T12:12:32.0358891Z ret = record(fn)(*args_) 2022-08-17T12:12:32.0359335Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-08-17T12:12:32.0359735Z return f(*args, **kwargs) 2022-08-17T12:12:32.0360030Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-08-17T12:12:32.0360389Z raise RuntimeError(msg) 2022-08-17T12:12:32.0360597Z RuntimeError: hello 2022-08-17T12:12:32.0360722Z 2022-08-17T12:12:32.0366981Z ok (1.524s) 2022-08-17T12:12:32.0389963Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s) 2022-08-17T12:12:32.0404284Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s) 2022-08-17T12:12:32.0459507Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 2021 via signal SIGTERM 2022-08-17T12:12:32.0486586Z ok (0.008s) 2022-08-17T12:12:32.0531357Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-08-17T12:12:32.0531985Z Traceback (most recent call last): 2022-08-17T12:12:32.0532752Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-08-17T12:12:32.0533343Z self._pc.join(-1) 2022-08-17T12:12:32.0533749Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-08-17T12:12:32.0534212Z return _mock_self._mock_call(*args, **kwargs) 2022-08-17T12:12:32.0534705Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-08-17T12:12:32.0535293Z raise effect 2022-08-17T12:12:32.0535762Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-08-17T12:12:32.0538583Z ok (0.005s) 2022-08-17T12:12:33.0967996Z test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:33.0968708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:33.0969664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:33.0970070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:34.4698234Z ok (2.416s) 2022-08-17T12:12:34.4782065Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 2032 closing signal SIGTERM 2022-08-17T12:12:34.4797533Z ok (0.010s) 2022-08-17T12:12:34.4819949Z test_to_map (__main__.StartProcessesListTest) ... ok (0.002s) 2022-08-17T12:12:34.4828096Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s) 2022-08-17T12:12:35.5651346Z test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:35.5652387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:35.5653176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:35.5653799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:35.5796062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:35.5796709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:35.5799026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:35.5799633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:35.7173122Z world 2022-08-17T12:12:35.7318869Z hello 2022-08-17T12:12:36.0243311Z Closing process 2033 via signal SIGTERM 2022-08-17T12:12:36.0267114Z ok (1.544s) 2022-08-17T12:12:36.0289282Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s) 2022-08-17T12:12:36.0621226Z test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1 2022-08-17T12:12:36.0621494Z bar stderr from 1 2022-08-17T12:12:36.1428084Z failed (exitcode: 138) local_rank: 0 (pid: 2053) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-08-17T12:12:36.1437465Z ok (0.115s) 2022-08-17T12:12:36.1524708Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.009s) 2022-08-17T12:12:36.1802162Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-08-17T12:12:36.1802831Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-08-17T12:12:36.1803336Z raise RuntimeError(f"raised from {rank}") 2022-08-17T12:12:36.1803758Z RuntimeError: raised from 0 2022-08-17T12:12:36.1844885Z bar from 1 2022-08-17T12:12:36.2651484Z failed (exitcode: 1) local_rank: 0 (pid: 2056) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-08-17T12:12:36.2660595Z ok (0.113s) 2022-08-17T12:12:37.3967461Z test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:37.3968186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:37.3969685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:37.3970380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:37.4270275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:37.4271571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:37.4272499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:37.4273163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:37.4439031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:37.4439602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:37.4441065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:37.4441667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:37.4480009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:37.4480683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:37.4483946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:37.4484819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:38.1136957Z Closing process 2058 via signal SIGTERM 2022-08-17T12:12:38.1137376Z Closing process 2059 via signal SIGTERM 2022-08-17T12:12:38.1137716Z Closing process 2060 via signal SIGTERM 2022-08-17T12:12:38.1355315Z ok (1.869s) 2022-08-17T12:12:38.1369532Z test_function_raise (__main__.StartProcessesTest) 2022-08-17T12:12:39.2417100Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:39.2417608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:39.2418567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:39.2419122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:39.2604468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:39.2604857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:39.2607413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:39.2608012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:39.6563085Z failed (exitcode: 1) local_rank: 0 (pid: 2098) of fn: echo2 (start_method: spawn) 2022-08-17T12:12:39.6563627Z Traceback (most recent call last): 2022-08-17T12:12:39.6564494Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-08-17T12:12:39.6565148Z self._pc.join(-1) 2022-08-17T12:12:39.6565827Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-08-17T12:12:39.6566634Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-08-17T12:12:39.6567021Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-08-17T12:12:39.6567243Z 2022-08-17T12:12:39.6567424Z -- Process 0 terminated with the following error: 2022-08-17T12:12:39.6567674Z Traceback (most recent call last): 2022-08-17T12:12:39.6568064Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-08-17T12:12:39.6568326Z fn(i, *args) 2022-08-17T12:12:39.6568723Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-08-17T12:12:39.6569032Z ret = record(fn)(*args_) 2022-08-17T12:12:39.6569449Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-08-17T12:12:39.6569978Z return f(*args, **kwargs) 2022-08-17T12:12:39.6570286Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-08-17T12:12:39.6570581Z raise RuntimeError(msg) 2022-08-17T12:12:39.6570770Z RuntimeError: hello 2022-08-17T12:12:39.6570897Z 2022-08-17T12:12:39.6576449Z ok (1.522s) 2022-08-17T12:12:39.6599259Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s) 2022-08-17T12:12:39.6613564Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s) 2022-08-17T12:12:39.6669535Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 2118 via signal SIGTERM 2022-08-17T12:12:39.6696706Z ok (0.008s) 2022-08-17T12:12:39.6734176Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-08-17T12:12:39.6734553Z Traceback (most recent call last): 2022-08-17T12:12:39.6735017Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-08-17T12:12:39.6735337Z self._pc.join(-1) 2022-08-17T12:12:39.6735596Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-08-17T12:12:39.6736038Z return _mock_self._mock_call(*args, **kwargs) 2022-08-17T12:12:39.6736311Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-08-17T12:12:39.6736553Z raise effect 2022-08-17T12:12:39.6736843Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-08-17T12:12:39.6739840Z ok (0.004s) 2022-08-17T12:12:40.7251254Z test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:40.7251981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:40.7252807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:40.7253414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:42.0868202Z ok (2.413s) 2022-08-17T12:12:42.0949892Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 2129 closing signal SIGTERM 2022-08-17T12:12:42.0965196Z ok (0.010s) 2022-08-17T12:12:42.0985641Z test_to_map (__main__.StartProcessesTest) ... ok (0.002s) 2022-08-17T12:12:42.0993801Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s) 2022-08-17T12:12:43.1711625Z test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:43.1712198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:43.1714043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:43.1714436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:43.2046916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:43.2047415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:43.2048284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:43.2048678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:43.3233503Z world 2022-08-17T12:12:43.3590285Z hello 2022-08-17T12:12:43.6474163Z Closing process 2130 via signal SIGTERM 2022-08-17T12:12:43.6523497Z ok (1.553s) 2022-08-17T12:12:43.6550103Z test_from_str_bad_input (__main__.StdTest) ... ok (0.002s) 2022-08-17T12:12:43.6561266Z test_from_value (__main__.StdTest) ... ok (0.001s) 2022-08-17T12:12:43.6570869Z test_from_value_map (__main__.StdTest) ... ok (0.001s) 2022-08-17T12:12:43.6571403Z 2022-08-17T12:12:43.6571950Z ---------------------------------------------------------------------- 2022-08-17T12:12:43.6572319Z Ran 38 tests in 18.301s 2022-08-17T12:12:43.6572442Z 2022-08-17T12:12:43.6572507Z OK 2022-08-17T12:12:43.6572605Z 2022-08-17T12:12:43.6572693Z Generating XML reports... 2022-08-17T12:12:43.6603642Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220817121225.xml 2022-08-17T12:12:43.6622191Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220817121225.xml 2022-08-17T12:12:43.6636778Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220817121225.xml 2022-08-17T12:12:43.6640620Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220817121225.xml 2022-08-17T12:12:43.9764001Z Running distributed/elastic/timer/api_test ... [2022-08-17 12:12:43.975985] 2022-08-17T12:12:43.9764784Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:43.976070] 2022-08-17T12:12:45.1264586Z Running distributed/elastic/timer/local_timer_example ... [2022-08-17 12:12:45.126004] 2022-08-17T12:12:45.1265450Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:45.126083] 2022-08-17T12:12:46.2395262Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-08-17T12:12:46.2406581Z 2022-08-17T12:12:46.2406686Z Running tests... 2022-08-17T12:12:46.2407126Z ---------------------------------------------------------------------- 2022-08-17T12:12:46.4981001Z test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-08-17 12:12:46,497 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-08-17T12:12:46.4981684Z [INFO] 2022-08-17 12:12:46,497 api: Starting watchdog thread... 2022-08-17T12:12:48.1234096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.1234807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.1249943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.1250635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.1460200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.1460870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.1461709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.1462374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2001401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2002076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2002919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2004482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2005298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2006075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2007679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2008575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2030768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2031855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2037779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2038757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2047422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2048368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2054015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2054656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2141921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2142616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2148243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2148861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.2597015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:48.2597694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:48.2603441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:48.2604111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:48.3952353Z [INFO] 2022-08-17 12:12:48,394 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.3967409Z [INFO] 2022-08-17 12:12:48,396 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4470245Z [INFO] 2022-08-17 12:12:48,446 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4500790Z [INFO] 2022-08-17 12:12:48,449 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4521708Z [INFO] 2022-08-17 12:12:48,451 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4526793Z [INFO] 2022-08-17 12:12:48,452 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4566583Z [INFO] 2022-08-17 12:12:48,456 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:48.4975342Z [INFO] 2022-08-17 12:12:48,496 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:49.5265001Z [INFO] 2022-08-17 12:12:49,525 api: Reaping worker_id=[2172]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:49.5265908Z [INFO] 2022-08-17 12:12:49,526 api: Successfully reaped worker=[2172] 2022-08-17T12:12:49.5774456Z [INFO] 2022-08-17 12:12:49,576 api: Reaping worker_id=[2176]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:49.5775168Z [INFO] 2022-08-17 12:12:49,577 api: Successfully reaped worker=[2176] 2022-08-17T12:12:49.5775845Z [INFO] 2022-08-17 12:12:49,577 api: Reaping worker_id=[2174]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:49.5776447Z [INFO] 2022-08-17 12:12:49,577 api: Successfully reaped worker=[2174] 2022-08-17T12:12:49.5877293Z [INFO] 2022-08-17 12:12:49,587 api: Reaping worker_id=[2178]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:49.5877767Z [INFO] 2022-08-17 12:12:49,587 api: Successfully reaped worker=[2178] 2022-08-17T12:12:49.5938332Z [INFO] 2022-08-17 12:12:49,593 api: Stopping LocalTimerServer 2022-08-17T12:12:49.5938676Z [INFO] 2022-08-17 12:12:49,593 api: Stopping watchdog thread... 2022-08-17T12:12:49.5984132Z ok (3.358s) 2022-08-17T12:12:49.6003897Z test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-08-17 12:12:49,600 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-08-17T12:12:49.6004357Z [INFO] 2022-08-17 12:12:49,600 api: Starting watchdog thread... 2022-08-17T12:12:51.2625381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.2626066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.2639587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.2640212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.2860160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.2860782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.2861562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.2862154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.2863231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.2863859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.2864680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.2865214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.3025006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.3026474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.3029986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.3031020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.3054133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.3054715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.3060931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.3061584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.3298064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.3298724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.3303091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.3303765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.3341182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.3341804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.3346955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.3347629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.3371452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:51.3372473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:51.3379241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:51.3380211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:51.5276960Z [INFO] 2022-08-17 12:12:51,527 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5367710Z [INFO] 2022-08-17 12:12:51,536 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5375198Z [INFO] 2022-08-17 12:12:51,537 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5522137Z [INFO] 2022-08-17 12:12:51,551 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5537642Z [INFO] 2022-08-17 12:12:51,553 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5755158Z [INFO] 2022-08-17 12:12:51,574 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5788010Z [INFO] 2022-08-17 12:12:51,578 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:51.5813187Z [INFO] 2022-08-17 12:12:51,580 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.1259272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.1259808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.1265693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.1266268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.1801442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.1802025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.1815085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.1815660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.1855572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.1856161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.1862579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.1863202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.1897995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.1898613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.1904724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.1905226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.2006424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.2007123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.2013121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.2013836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.2183357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.2184043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.2190991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.2191618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.2362948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.2363921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.2367747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.2368393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.2401746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:54.2402380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:54.2408661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:54.2409336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:54.3847347Z [INFO] 2022-08-17 12:12:54,384 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4327847Z [INFO] 2022-08-17 12:12:54,432 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4375727Z [INFO] 2022-08-17 12:12:54,437 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4411701Z [INFO] 2022-08-17 12:12:54,440 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4454676Z [INFO] 2022-08-17 12:12:54,445 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4645282Z [INFO] 2022-08-17 12:12:54,463 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4781478Z [INFO] 2022-08-17 12:12:54,477 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:54.4805003Z [INFO] 2022-08-17 12:12:54,480 api: Timer client configured to: LocalTimerClient 2022-08-17T12:12:55.5164508Z [INFO] 2022-08-17 12:12:55,515 api: Reaping worker_id=[2352]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:55.5165053Z [INFO] 2022-08-17 12:12:55,516 api: Successfully reaped worker=[2352] 2022-08-17T12:12:55.5571729Z [INFO] 2022-08-17 12:12:55,556 api: Reaping worker_id=[2356]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:55.5572199Z [INFO] 2022-08-17 12:12:55,556 api: Successfully reaped worker=[2356] 2022-08-17T12:12:55.5655499Z [INFO] 2022-08-17 12:12:55,565 api: Stopping LocalTimerServer 2022-08-17T12:12:55.5655999Z [INFO] 2022-08-17 12:12:55,565 api: Stopping watchdog thread... 2022-08-17T12:12:55.5673181Z [INFO] 2022-08-17 12:12:55,567 api: Reaping worker_id=[2354]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:55.5673815Z [INFO] 2022-08-17 12:12:55,567 local_timer: Process with pid=2354 does not exist. Skipping 2022-08-17T12:12:55.5674229Z [INFO] 2022-08-17 12:12:55,567 api: Successfully reaped worker=[2354] 2022-08-17T12:12:55.5674679Z [INFO] 2022-08-17 12:12:55,567 api: Reaping worker_id=[2353]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-08-17T12:12:55.5675207Z [INFO] 2022-08-17 12:12:55,567 local_timer: Process with pid=2353 does not exist. Skipping 2022-08-17T12:12:55.5675608Z [INFO] 2022-08-17 12:12:55,567 api: Successfully reaped worker=[2353] 2022-08-17T12:12:55.5679090Z ok (5.969s) 2022-08-17T12:12:55.5681037Z 2022-08-17T12:12:55.5681522Z ---------------------------------------------------------------------- 2022-08-17T12:12:55.5681816Z Ran 2 tests in 9.327s 2022-08-17T12:12:55.5681955Z 2022-08-17T12:12:55.5682016Z OK 2022-08-17T12:12:55.5682110Z 2022-08-17T12:12:55.5682207Z Generating XML reports... 2022-08-17T12:12:55.5716555Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220817121246.xml 2022-08-17T12:12:55.8985349Z Running distributed/elastic/timer/local_timer_test ... [2022-08-17 12:12:55.898155] 2022-08-17T12:12:55.8985967Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:12:55.898237] 2022-08-17T12:12:57.0063774Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-08-17T12:12:57.0076243Z 2022-08-17T12:12:57.0076351Z Running tests... 2022-08-17T12:12:57.0076829Z ---------------------------------------------------------------------- 2022-08-17T12:12:57.0084907Z test_acquire_release (__main__.LocalTimerServerTest) 2022-08-17T12:12:57.2641299Z tests that: ... ok (0.256s) 2022-08-17T12:12:57.2647293Z test_expired_timers (__main__.LocalTimerServerTest) 2022-08-17T12:12:57.2662039Z tests that a single expired timer on a process should terminate ... ok (0.002s) 2022-08-17T12:12:57.2672393Z test_valid_timers (__main__.LocalTimerServerTest) 2022-08-17T12:12:57.2686468Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s) 2022-08-17T12:12:57.2692120Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-08-17T12:12:57.3716309Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.103s) 2022-08-17T12:12:57.3717538Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-08-17T12:12:57.3826888Z checks that the watchdog can run on an empty queue ... ok (0.011s) 2022-08-17T12:12:57.3898402Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.007s) 2022-08-17T12:12:57.4014407Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s) 2022-08-17T12:12:57.4022332Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-08-17T12:12:58.6337594Z If a function acquires a countdown timer with default scope, ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:12:58.6338157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:12:58.6340757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:12:58.6341295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:12:58.9602017Z ok (1.558s) 2022-08-17T12:12:59.0635734Z test_happy_path (__main__.LocalTimerTest) ... ok (0.103s) 2022-08-17T12:12:59.0749777Z test_no_client (__main__.LocalTimerTest) ... ok (0.011s) 2022-08-17T12:12:59.2042687Z test_timer (__main__.LocalTimerTest) ... ok (0.129s) 2022-08-17T12:12:59.2271349Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.022s) 2022-08-17T12:12:59.2278363Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-08-17T12:12:59.7345893Z Tests slow producer. ... ok (0.507s) 2022-08-17T12:12:59.7358476Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-08-17T12:13:00.6446393Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.910s) 2022-08-17T12:13:00.6447722Z 2022-08-17T12:13:00.6448203Z ---------------------------------------------------------------------- 2022-08-17T12:13:00.6448689Z Ran 14 tests in 3.637s 2022-08-17T12:13:00.6448895Z 2022-08-17T12:13:00.6448950Z OK 2022-08-17T12:13:00.6449047Z 2022-08-17T12:13:00.6449139Z Generating XML reports... 2022-08-17T12:13:00.6512578Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220817121257.xml 2022-08-17T12:13:00.6519094Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220817121257.xml 2022-08-17T12:13:00.6524746Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220817121257.xml 2022-08-17T12:13:01.0753588Z Running distributed/elastic/utils/distributed_test ... [2022-08-17 12:13:01.074951] 2022-08-17T12:13:01.0754229Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:01.075039] 2022-08-17T12:13:02.2568319Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-08-17T12:13:02.2579891Z 2022-08-17T12:13:02.2580301Z Running tests... 2022-08-17T12:13:02.2580944Z ---------------------------------------------------------------------- 2022-08-17T12:13:02.5297789Z test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.271s) 2022-08-17T12:13:02.5309373Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s) 2022-08-17T12:13:02.5314751Z test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.000s) 2022-08-17T12:13:05.5643886Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.033s) 2022-08-17T12:13:05.5653101Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (b92e618f4d32, 0). 2022-08-17T12:13:05.5654611Z ok (0.001s) 2022-08-17T12:13:05.5669102Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:39321 (errno: 98 - Address already in use). 2022-08-17T12:13:05.5682132Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:39321 (errno: 98 - Address already in use). 2022-08-17T12:13:05.5682572Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-08-17T12:13:05.5684373Z ok (0.003s) 2022-08-17T12:13:05.5708937Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (b92e618f4d32, 38139). 2022-08-17T12:13:05.5710317Z ok (0.003s) 2022-08-17T12:13:05.5710882Z 2022-08-17T12:13:05.5711231Z ---------------------------------------------------------------------- 2022-08-17T12:13:05.5711700Z Ran 7 tests in 3.313s 2022-08-17T12:13:05.5711829Z 2022-08-17T12:13:05.5711908Z OK (skipped=1) 2022-08-17T12:13:05.5712008Z 2022-08-17T12:13:05.5712099Z Generating XML reports... 2022-08-17T12:13:05.5747240Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220817121302.xml 2022-08-17T12:13:05.8993029Z Running distributed/elastic/utils/logging_test ... [2022-08-17 12:13:05.898887] 2022-08-17T12:13:05.8993650Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:05.898971] 2022-08-17T12:13:07.0746134Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test 2022-08-17T12:13:07.0759573Z 2022-08-17T12:13:07.0759951Z Running tests... 2022-08-17T12:13:07.0760371Z ---------------------------------------------------------------------- 2022-08-17T12:13:07.3509333Z test_derive_module_name (__main__.LoggingTest) ... ok (0.275s) 2022-08-17T12:13:07.3528267Z test_logger_name (__main__.LoggingTest) ... ok (0.002s) 2022-08-17T12:13:07.3528669Z 2022-08-17T12:13:07.3529200Z ---------------------------------------------------------------------- 2022-08-17T12:13:07.3529534Z Ran 2 tests in 0.277s 2022-08-17T12:13:07.3529657Z 2022-08-17T12:13:07.3529730Z OK 2022-08-17T12:13:07.3529827Z 2022-08-17T12:13:07.3529921Z Generating XML reports... 2022-08-17T12:13:07.3555829Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220817121307.xml 2022-08-17T12:13:07.6455387Z Running distributed/elastic/utils/util_test ... [2022-08-17 12:13:07.645127] 2022-08-17T12:13:07.6456001Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:07.645213] 2022-08-17T12:13:08.7381887Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-08-17T12:13:08.7393841Z 2022-08-17T12:13:08.7394010Z Running tests... 2022-08-17T12:13:08.7394643Z ---------------------------------------------------------------------- 2022-08-17T12:13:08.9984959Z test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.259s) 2022-08-17T12:13:09.0001057Z test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s) 2022-08-17T12:13:09.0028374Z test_synchronize (__main__.StoreUtilTest) ... ok (0.003s) 2022-08-17T12:13:09.0802737Z test_get_logger (__main__.UtilTest) ... ok (0.077s) 2022-08-17T12:13:09.0810752Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-08-17T12:13:09.0820056Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-08-17T12:13:09.0831306Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-08-17T12:13:09.0831662Z 2022-08-17T12:13:09.0832142Z ---------------------------------------------------------------------- 2022-08-17T12:13:09.0832587Z Ran 7 tests in 0.344s 2022-08-17T12:13:09.0832789Z 2022-08-17T12:13:09.0832855Z OK 2022-08-17T12:13:09.0832952Z 2022-08-17T12:13:09.0833045Z Generating XML reports... 2022-08-17T12:13:09.0861100Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220817121308.xml 2022-08-17T12:13:09.0867078Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220817121308.xml 2022-08-17T12:13:09.3824456Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-08-17 12:13:09.381951] 2022-08-17T12:13:09.3825130Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:09.382038] 2022-08-17T12:13:10.4938868Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper 2022-08-17T12:13:10.4950698Z 2022-08-17T12:13:10.4951146Z Running tests... 2022-08-17T12:13:10.4951631Z ---------------------------------------------------------------------- 2022-08-17T12:13:10.4975012Z test_apply_activation_checkpointing_wrapper (__main__.CheckpointWrapperTest) 2022-08-17T12:13:10.7948978Z Ensures that `apply_activation_checkpointing_wrapper` can be used ... ok (0.299s) 2022-08-17T12:13:10.7968636Z test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest) 2022-08-17T12:13:10.7969777Z Tests that using checkpoint_wrapper or the functional ... skip: Test requires CUDA (0.002s) 2022-08-17T12:13:10.7985372Z test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-08-17T12:13:10.7995682Z test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-08-17T12:13:10.8025655Z test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s) 2022-08-17T12:13:10.8025937Z 2022-08-17T12:13:10.8026903Z ---------------------------------------------------------------------- 2022-08-17T12:13:10.8027204Z Ran 5 tests in 0.308s 2022-08-17T12:13:10.8027354Z 2022-08-17T12:13:10.8027833Z OK (skipped=1) 2022-08-17T12:13:10.8028005Z 2022-08-17T12:13:10.8028121Z Generating XML reports... 2022-08-17T12:13:10.8059710Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20220817121310.xml 2022-08-17T12:13:11.1109557Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-08-17 12:13:11.110546] 2022-08-17T12:13:11.1110185Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:11.110630] 2022-08-17T12:13:12.2330857Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint 2022-08-17T12:13:12.2343275Z 2022-08-17T12:13:12.2343598Z Running tests... 2022-08-17T12:13:12.2344217Z ---------------------------------------------------------------------- 2022-08-17T12:13:12.4946573Z test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2544 2022-08-17T12:13:12.4974488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2545 2022-08-17T12:13:13.4743014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:13.4743688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:13.4745455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:13.4746058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:13.4785690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:13.4786330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:13.4790185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:13.4790827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:13.6278425Z dist init r=0, world=2 2022-08-17T12:13:13.6332259Z dist init r=1, world=2 2022-08-17T12:13:13.6444658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:13.6445103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:13.6446137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:13.6446714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:13.6450925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:13.6451322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:13.9007910Z skip: Need at least 2 CUDA devices (1.666s) 2022-08-17T12:13:13.9069548Z test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2572 2022-08-17T12:13:13.9106493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2573 2022-08-17T12:13:14.9148351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:14.9148829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:14.9151675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:14.9152076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:14.9382317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:14.9382710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:14.9385809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:14.9386174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:15.0695073Z dist init r=1, world=2 2022-08-17T12:13:15.0938660Z dist init r=0, world=2 2022-08-17T12:13:15.1107663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:15.1108280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:15.1108929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:15.1109492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:15.1114833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:15.1115512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:15.4137953Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:13:15.4138212Z 2022-08-17T12:13:15.4138587Z ---------------------------------------------------------------------- 2022-08-17T12:13:15.4138978Z Ran 2 tests in 3.179s 2022-08-17T12:13:15.4139160Z 2022-08-17T12:13:15.4139284Z OK (skipped=2) 2022-08-17T12:13:15.4139568Z 2022-08-17T12:13:15.4139686Z Generating XML reports... 2022-08-17T12:13:15.4177631Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220817121312.xml 2022-08-17T12:13:15.7427019Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-08-17 12:13:15.742287] 2022-08-17T12:13:15.7427676Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:15.742373] 2022-08-17T12:13:16.8759186Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper 2022-08-17T12:13:16.8772397Z 2022-08-17T12:13:16.8772900Z Running tests... 2022-08-17T12:13:16.8773300Z ---------------------------------------------------------------------- 2022-08-17T12:13:17.1281632Z test_empty_module (__main__.TestFlattenParams) ... ok (0.251s) 2022-08-17T12:13:17.1347064Z test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.007s) 2022-08-17T12:13:17.1419616Z test_num_params (__main__.TestFlattenParams) ... ok (0.007s) 2022-08-17T12:13:17.1601534Z test_output (__main__.TestFlattenParams) ... ok (0.018s) 2022-08-17T12:13:17.1694826Z test_partial_flattening (__main__.TestFlattenParams) ... ok (0.009s) 2022-08-17T12:13:17.1768982Z test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.007s) 2022-08-17T12:13:17.1848402Z test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.008s) 2022-08-17T12:13:17.2003049Z test_shared_params_output (__main__.TestFlattenParams) ... ok (0.015s) 2022-08-17T12:13:17.2287168Z test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.028s) 2022-08-17T12:13:17.2293372Z test_empty_module (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s) 2022-08-17T12:13:17.2297322Z test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2300831Z test_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2304366Z test_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2320117Z test_partial_flattening (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.002s) 2022-08-17T12:13:17.2355775Z test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.004s) 2022-08-17T12:13:17.2359621Z test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2362834Z test_shared_params_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2368713Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s) 2022-08-17T12:13:17.2374766Z test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.001s) 2022-08-17T12:13:17.2378489Z test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2381410Z test_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2385073Z test_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2401125Z test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.002s) 2022-08-17T12:13:17.2436472Z test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.003s) 2022-08-17T12:13:17.2439760Z test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2442801Z test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2448492Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s) 2022-08-17T12:13:17.2448900Z 2022-08-17T12:13:17.2449351Z ---------------------------------------------------------------------- 2022-08-17T12:13:17.2449731Z Ran 27 tests in 0.368s 2022-08-17T12:13:17.2449909Z 2022-08-17T12:13:17.2450025Z OK (skipped=18) 2022-08-17T12:13:17.2450206Z 2022-08-17T12:13:17.2450347Z Generating XML reports... 2022-08-17T12:13:17.2482036Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220817121316.xml 2022-08-17T12:13:17.2493603Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220817121316.xml 2022-08-17T12:13:17.2505518Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220817121316.xml 2022-08-17T12:13:17.5426409Z Running distributed/fsdp/test_fsdp_apply ... [2022-08-17 12:13:17.542189] 2022-08-17T12:13:17.5427063Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:17.542274] 2022-08-17T12:13:18.6501632Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2022-08-17T12:13:18.6514341Z 2022-08-17T12:13:18.6514772Z Running tests... 2022-08-17T12:13:18.6515209Z ---------------------------------------------------------------------- 2022-08-17T12:13:18.6521167Z test_apply_in_summon_raises_error (__main__.TestApply) 2022-08-17T12:13:18.9176709Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2624 2022-08-17T12:13:18.9204719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2625 2022-08-17T12:13:19.9062433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:19.9062852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:19.9064526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:19.9064963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:19.9150359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:19.9150737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:19.9153922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:19.9154295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:20.0639727Z dist init r=1, world=2 2022-08-17T12:13:20.0703624Z dist init r=0, world=2 2022-08-17T12:13:20.0850394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:20.0850824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:20.0853772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:20.0854478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:20.0860524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:20.0861124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:20.4239242Z skip: Need at least 2 CUDA devices (1.772s) 2022-08-17T12:13:20.4247194Z test_nested_module_apply (__main__.TestApply) 2022-08-17T12:13:20.4296199Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2652 2022-08-17T12:13:20.4335596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2653 2022-08-17T12:13:21.4087327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:21.4087893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:21.4088399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:21.4088823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:21.4089761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:21.4090728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:21.4091536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:21.4091918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:21.5650699Z dist init r=1, world=2 2022-08-17T12:13:21.5651074Z dist init r=0, world=2 2022-08-17T12:13:21.5760144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:21.5760903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:21.5764317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:21.5865691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:21.5870485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:21.5871097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:21.8366858Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:13:21.8371182Z test_transformer_module_apply (__main__.TestApply) 2022-08-17T12:13:21.8419049Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2680 2022-08-17T12:13:21.8456828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2681 2022-08-17T12:13:22.8271098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:22.8271594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:22.8273833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:22.8274340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:22.8288217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:22.8288693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:22.8291598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:22.8292208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:22.9852315Z dist init r=1, world=2 2022-08-17T12:13:22.9852829Z dist init r=0, world=2 2022-08-17T12:13:23.0062116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:23.0062726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:23.0065182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:23.0065872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:13:23.0170893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:23.0171426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:23.3487262Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:13:23.3487536Z 2022-08-17T12:13:23.3487990Z ---------------------------------------------------------------------- 2022-08-17T12:13:23.3488404Z Ran 3 tests in 4.697s 2022-08-17T12:13:23.3488580Z 2022-08-17T12:13:23.3488658Z OK (skipped=3) 2022-08-17T12:13:23.3488759Z 2022-08-17T12:13:23.3488851Z Generating XML reports... 2022-08-17T12:13:23.3525803Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220817121318.xml 2022-08-17T12:13:23.6810689Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-08-17 12:13:23.680675] 2022-08-17T12:13:23.6811330Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:23.680764] 2022-08-17T12:13:24.8235905Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-08-17T12:13:24.8248051Z 2022-08-17T12:13:24.8248375Z Running tests... 2022-08-17T12:13:24.8249065Z ---------------------------------------------------------------------- 2022-08-17T12:13:25.0971555Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2719 2022-08-17T12:13:25.1000967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2720 2022-08-17T12:13:25.1028634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2721 2022-08-17T12:13:25.1057779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2722 2022-08-17T12:13:26.2043604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:26.2044283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:26.2045429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:26.2046277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:26.2278170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:26.2278880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:26.2280245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:26.2280926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:26.2391010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:26.2391683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:26.2394385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:26.2395039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:26.2584927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:26.2585863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:26.2587339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:26.2588060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:26.3628235Z dist init r=0, world=4 2022-08-17T12:13:26.3849807Z dist init r=2, world=4 2022-08-17T12:13:26.3981461Z dist init r=1, world=4 2022-08-17T12:13:26.4170837Z dist init r=3, world=4 2022-08-17T12:13:26.4281228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:26.4382713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:26.4395867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:26.4396333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:26.4397018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:26.4397767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:26.4485656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:26.4486582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:26.4503882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:26.4504568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:26.4505142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:26.4505700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:26.8104451Z skip: Need at least 2 CUDA devices (1.985s) 2022-08-17T12:13:26.8170220Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2775 2022-08-17T12:13:26.8208472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2776 2022-08-17T12:13:26.8238807Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2777 2022-08-17T12:13:26.8270017Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2778 2022-08-17T12:13:27.8863544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:27.8864188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:27.8865351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:27.8865981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:27.8923277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:27.8923951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:27.8926955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:27.8927611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:27.8972785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:27.8973398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:27.8976126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:27.8977049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:27.9060322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:27.9060965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:27.9062411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:27.9063036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:28.0466205Z dist init r=1, world=4 2022-08-17T12:13:28.0549196Z dist init r=2, world=4 2022-08-17T12:13:28.0573271Z dist init r=3, world=4 2022-08-17T12:13:28.0736666Z dist init r=0, world=4 2022-08-17T12:13:28.0887201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:28.0986933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:28.1088825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:28.1089462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:28.1090124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:28.1090685Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:28.1091224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:28.1091765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:28.1197053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:28.1198364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:28.1198978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:28.1199539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:28.4310538Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:28.4377764Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2831 2022-08-17T12:13:28.4415475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2832 2022-08-17T12:13:28.4446987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2833 2022-08-17T12:13:28.4477403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2834 2022-08-17T12:13:29.4484370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:29.4485067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:29.4486830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:29.4487489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:29.4818890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:29.4819625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:29.4820439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:29.4821084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:29.4950863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:29.4951625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:29.4953898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:29.4954583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:29.5168252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:29.5168962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:29.5169790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:29.5170381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:29.6082827Z dist init r=3, world=4 2022-08-17T12:13:29.6422756Z dist init r=1, world=4 2022-08-17T12:13:29.6560318Z dist init r=2, world=4 2022-08-17T12:13:29.6735062Z dist init r=0, world=4 2022-08-17T12:13:29.7148620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:29.7249815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:29.7351700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:29.7352353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:29.7353326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:29.7354168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:29.7355013Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:29.7355850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:29.7359779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:29.7360557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:29.7361318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:29.7362293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:30.0517449Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:13:30.0584066Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2887 2022-08-17T12:13:30.0622260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2888 2022-08-17T12:13:30.0652689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2889 2022-08-17T12:13:30.0682192Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2890 2022-08-17T12:13:31.1292856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:31.1293643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:31.1295244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:31.1295881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:31.1415717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:31.1416688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:31.1418863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:31.1419497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:31.1563639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:31.1564299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:31.1565780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:31.1566596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:31.1864029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:31.1864623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:31.1868042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:31.1868793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:31.3102003Z dist init r=2, world=4 2022-08-17T12:13:31.3171473Z dist init r=0, world=4 2022-08-17T12:13:31.3287300Z dist init r=1, world=4 2022-08-17T12:13:31.3446067Z dist init r=3, world=4 2022-08-17T12:13:31.3556290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:31.3615147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:31.3716827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:31.3717256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:31.3717913Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:31.3718456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:31.3719017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:31.3759896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:31.3825755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:31.3826375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:31.3826943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:31.3827538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:31.6723397Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:31.6785830Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2943 2022-08-17T12:13:31.6822669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2944 2022-08-17T12:13:31.6851158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2945 2022-08-17T12:13:31.6887076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2946 2022-08-17T12:13:32.7050236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:32.7050944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:32.7052159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:32.7053093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:32.7120685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:32.7121359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:32.7123814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:32.7124499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:32.7371121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:32.7371804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:32.7373127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:32.7373804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:32.7404110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:32.7404767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:32.7407308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:32.7407925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:32.8650808Z dist init r=3, world=4 2022-08-17T12:13:32.8738298Z dist init r=0, world=4 2022-08-17T12:13:32.8968149Z dist init r=2, world=4 2022-08-17T12:13:32.8998145Z dist init r=1, world=4 2022-08-17T12:13:32.9311679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:32.9312095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:32.9413069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:32.9413942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:32.9415442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:32.9416223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:32.9416922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:32.9417728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:32.9421871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:32.9425014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:32.9425602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:32.9426214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:33.2925540Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:33.2989737Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2999 2022-08-17T12:13:33.3038724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3000 2022-08-17T12:13:33.3075133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3001 2022-08-17T12:13:33.3114313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3002 2022-08-17T12:13:34.2882021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:34.2882605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:34.2883747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:34.2884384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:34.2944302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:34.2944914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:34.2947672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:34.2948326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:34.3022385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:34.3023019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:34.3025408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:34.3026314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:34.3421378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:34.3422121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:34.3424510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:34.3425262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:34.4517626Z dist init r=3, world=4 2022-08-17T12:13:34.4591556Z dist init r=1, world=4 2022-08-17T12:13:34.4814616Z dist init r=0, world=4 2022-08-17T12:13:34.5047365Z dist init r=2, world=4 2022-08-17T12:13:34.5308737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:34.5409552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:34.5512296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:34.5513217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:34.5513722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:34.5514526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:34.5515285Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:34.5515918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:34.5520441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:34.5521102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:34.5521611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:34.5521990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:34.9153915Z skip: Need at least 2 CUDA devices (1.623s) 2022-08-17T12:13:34.9216190Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3055 2022-08-17T12:13:34.9253975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3056 2022-08-17T12:13:34.9283735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3057 2022-08-17T12:13:34.9313690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3058 2022-08-17T12:13:35.9166983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:35.9167646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:35.9168798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:35.9169442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:35.9658268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:35.9658896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:35.9660424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:35.9661078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:35.9767465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:35.9768064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:35.9769314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:35.9769896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:35.9831869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:35.9832519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:35.9835207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:35.9835897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:36.0739761Z dist init r=2, world=4 2022-08-17T12:13:36.1256544Z dist init r=3, world=4 2022-08-17T12:13:36.1356755Z dist init r=0, world=4 2022-08-17T12:13:36.1430617Z dist init r=1, world=4 2022-08-17T12:13:36.1669427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:36.1669856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:36.1773372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:36.1773978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:36.1774937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:36.1775762Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:36.1776394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:36.1777145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:36.1778064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:36.1778540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:36.1779086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:36.1779653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:36.5354306Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:36.5372622Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s) 2022-08-17T12:13:36.5373211Z 2022-08-17T12:13:36.5373435Z ---------------------------------------------------------------------- 2022-08-17T12:13:36.5373696Z Ran 8 tests in 11.712s 2022-08-17T12:13:36.5373819Z 2022-08-17T12:13:36.5373881Z OK (skipped=8) 2022-08-17T12:13:36.5373994Z 2022-08-17T12:13:36.5374081Z Generating XML reports... 2022-08-17T12:13:36.5414879Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220817121324.xml 2022-08-17T12:13:36.8757498Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-08-17 12:13:36.875352] 2022-08-17T12:13:36.8758115Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:13:36.875438] 2022-08-17T12:13:38.0382641Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-08-17T12:13:38.0395431Z 2022-08-17T12:13:38.0395607Z Running tests... 2022-08-17T12:13:38.0396135Z ---------------------------------------------------------------------- 2022-08-17T12:13:38.0404423Z test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-08-17T12:13:38.3098001Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3122 2022-08-17T12:13:38.3127046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3123 2022-08-17T12:13:38.3156168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3124 2022-08-17T12:13:38.3188225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3125 2022-08-17T12:13:39.3809982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:39.3810695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:39.3812505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:39.3813176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:39.3930917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:39.3931538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:39.3933010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:39.3933603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:39.3942809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:39.3943489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:39.3946455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:39.3947135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:39.3999258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:39.3999913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:39.4001517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:39.4002155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:39.5423307Z dist init r=3, world=4 2022-08-17T12:13:39.5585045Z dist init r=0, world=4 2022-08-17T12:13:39.5587197Z dist init r=1, world=4 2022-08-17T12:13:39.5718751Z dist init r=2, world=4 2022-08-17T12:13:39.6001296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:39.6101720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:39.6204212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:39.6204941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:39.6205638Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:39.6206638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:39.6207216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:39.6207929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:39.6312536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:39.6313132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:39.6313727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:39.6314157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:39.9233903Z skip: Need at least 2 CUDA devices (1.884s) 2022-08-17T12:13:39.9241600Z test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-08-17T12:13:39.9290063Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3178 2022-08-17T12:13:39.9325413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3179 2022-08-17T12:13:39.9353902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3180 2022-08-17T12:13:39.9382614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3181 2022-08-17T12:13:41.0035564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:41.0036206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:41.0037956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:41.0038620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:41.0381293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:41.0381853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:41.0383125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:41.0383692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:41.0655915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:41.0656576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:41.0657688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:41.0658316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:41.1141418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:41.1142410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:41.1143300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:41.1143976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:41.1633778Z dist init r=1, world=4 2022-08-17T12:13:41.1982302Z dist init r=0, world=4 2022-08-17T12:13:41.2247531Z dist init r=2, world=4 2022-08-17T12:13:41.2736761Z dist init r=3, world=4 2022-08-17T12:13:41.2851505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:41.2852373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:41.2853020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:41.2854006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:41.2854685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:41.2855282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:41.2855816Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:41.2856484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:41.2861178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:41.2861854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:41.2862428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:41.2863235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:41.6425203Z skip: Need at least 2 CUDA devices (1.719s) 2022-08-17T12:13:41.6433042Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-08-17T12:13:41.6478653Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3234 2022-08-17T12:13:41.6515020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3235 2022-08-17T12:13:41.6546147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3236 2022-08-17T12:13:41.6585248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3237 2022-08-17T12:13:42.6392227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:42.6392900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:42.6394718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:42.6395335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:42.6397453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:42.6398002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:42.6402814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:42.6403472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:42.6625086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:42.6625960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:42.6627429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:42.6628285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:42.7171935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:42.7172584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:42.7173919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:42.7174559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:42.8036034Z dist init r=2, world=4 2022-08-17T12:13:42.8158194Z dist init r=0, world=4 2022-08-17T12:13:42.8274788Z dist init r=1, world=4 2022-08-17T12:13:42.8762744Z dist init r=3, world=4 2022-08-17T12:13:42.8870927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:42.8889275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:42.8889781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:42.8890424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:42.8891086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:42.8891639Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:42.8892191Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:42.8973713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:42.8997164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:42.8997845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:42.8998465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:42.8999028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:43.2629961Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:43.2638091Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-08-17T12:13:43.2686640Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3290 2022-08-17T12:13:43.2726862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3291 2022-08-17T12:13:43.2757941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3292 2022-08-17T12:13:43.2787341Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3293 2022-08-17T12:13:44.2889236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:44.2889890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:44.2891257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:44.2891901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:44.3035378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:44.3036021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:44.3038329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:44.3038984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:44.3459626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:44.3460270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:44.3461076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:44.3461720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:44.3491609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:44.3492268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:44.3495797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:44.3496501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:44.4483602Z dist init r=2, world=4 2022-08-17T12:13:44.4626872Z dist init r=1, world=4 2022-08-17T12:13:44.5037967Z dist init r=3, world=4 2022-08-17T12:13:44.5094486Z dist init r=0, world=4 2022-08-17T12:13:44.5247966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:44.5441638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:44.5442129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:44.5442591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:44.5443297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:44.5443846Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:44.5444477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:44.5451137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:44.5550355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:44.5551020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:44.5551596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:44.5551968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:44.8826689Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:44.8834181Z test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-08-17T12:13:44.8880725Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3346 2022-08-17T12:13:44.8919526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3347 2022-08-17T12:13:44.8949172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3348 2022-08-17T12:13:44.8986496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3349 2022-08-17T12:13:45.9086612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:45.9087279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:45.9088550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:45.9089202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:45.9230384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:45.9231417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:45.9232993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:45.9233640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:45.9555061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:45.9555712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:45.9556976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:45.9557630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:45.9566068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:45.9566701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:45.9569865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:45.9570501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:46.0673703Z dist init r=2, world=4 2022-08-17T12:13:46.0820095Z dist init r=3, world=4 2022-08-17T12:13:46.1118694Z dist init r=1, world=4 2022-08-17T12:13:46.1132368Z dist init r=0, world=4 2022-08-17T12:13:46.1329739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:46.1529059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:46.1631260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:46.1632116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:46.1632781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:46.1633341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:46.1634010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:46.1634561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:46.1738666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:46.1739265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:46.1739837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:46.1740238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:46.5026820Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:46.5034496Z test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-08-17T12:13:46.5082786Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3402 2022-08-17T12:13:46.5122553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3403 2022-08-17T12:13:46.5152269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3404 2022-08-17T12:13:46.5181526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3405 2022-08-17T12:13:47.5046899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:47.5047775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:47.5048623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:47.5049512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:47.5050338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:47.5050988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:47.5051797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:47.5052453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:47.5267070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:47.5267740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:47.5270002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:47.5270683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:47.5570952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:47.5571830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:47.5573877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:47.5574406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:47.6635240Z dist init r=1, world=4 2022-08-17T12:13:47.6638317Z dist init r=0, world=4 2022-08-17T12:13:47.6847781Z dist init r=3, world=4 2022-08-17T12:13:47.7124436Z dist init r=2, world=4 2022-08-17T12:13:47.7257128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:47.7347903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:47.7449953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:47.7450588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:47.7451271Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:47.7451828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:47.7452382Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:47.7461069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:47.7557366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:47.7557976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:47.7558459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:47.7559024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:48.1221359Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:48.1228817Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-08-17T12:13:48.1278001Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3458 2022-08-17T12:13:48.1316940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3459 2022-08-17T12:13:48.1346957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3460 2022-08-17T12:13:48.1377051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3461 2022-08-17T12:13:49.1521681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:49.1522446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:49.1523671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:49.1524318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:49.1718181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:49.1718865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:49.1720202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:49.1720897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:49.1949632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:49.1950332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:49.1951227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:49.1952103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:49.1955183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:49.1955903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:49.1959787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:49.1960491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:49.3159684Z dist init r=3, world=4 2022-08-17T12:13:49.3337816Z dist init r=1, world=4 2022-08-17T12:13:49.3567978Z dist init r=2, world=4 2022-08-17T12:13:49.3579716Z dist init r=0, world=4 2022-08-17T12:13:49.3951584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:49.4051922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:49.4153882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:49.4154502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:49.4155316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:49.4156109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:49.4156720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:49.4157284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:49.4165034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:49.4166998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:49.4167700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:49.4168116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:49.7417593Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:49.7425507Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-08-17T12:13:49.7475526Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3514 2022-08-17T12:13:49.7515083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3515 2022-08-17T12:13:49.7544809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3516 2022-08-17T12:13:49.7575435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3517 2022-08-17T12:13:50.7710425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:50.7711080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:50.7711986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:50.7712640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:50.7941492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:50.7943036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:50.7943911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:50.7944534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:50.8277553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:50.8278310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:50.8279090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:50.8279716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:50.8280490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:50.8281066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:50.8281865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:50.8282471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:50.9323273Z dist init r=0, world=4 2022-08-17T12:13:50.9530680Z dist init r=2, world=4 2022-08-17T12:13:50.9864836Z dist init r=3, world=4 2022-08-17T12:13:50.9865356Z dist init r=1, world=4 2022-08-17T12:13:51.0143339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:51.0245015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:51.0245606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:51.0246314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:51.0247096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:51.0247778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:51.0248339Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:51.0248876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:51.0354918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:51.0355326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:51.0355699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:51.0356050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:51.3615674Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:13:51.3622942Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-08-17T12:13:51.3671185Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3570 2022-08-17T12:13:51.3709114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3571 2022-08-17T12:13:51.3738110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3572 2022-08-17T12:13:51.3767847Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3573 2022-08-17T12:13:52.3755053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:52.3755721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:52.3756939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:52.3757601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:52.3854201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:52.3855099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:52.3856630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:52.3857280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:52.4303573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:52.4304128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:52.4305575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:52.4306272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:52.4378433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:52.4379088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:52.4382046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:52.4382690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:52.5337860Z dist init r=2, world=4 2022-08-17T12:13:52.5443274Z dist init r=3, world=4 2022-08-17T12:13:52.5886632Z dist init r=0, world=4 2022-08-17T12:13:52.5966168Z dist init r=1, world=4 2022-08-17T12:13:52.6298438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:52.6398916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:52.6501818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:52.6502889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:52.6503685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:52.6504631Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:52.6505188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:52.6505836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:52.6509497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:52.6510149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:52.6510689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:52.6512450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:52.9808732Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:52.9814732Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-08-17T12:13:52.9862226Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3626 2022-08-17T12:13:52.9900104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3627 2022-08-17T12:13:52.9928858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3628 2022-08-17T12:13:52.9958831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3629 2022-08-17T12:13:54.0095436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:54.0096138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:54.0097373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:54.0098013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:54.0228484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:54.0229147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:54.0230092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:54.0230699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:54.0453225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:54.0453976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:54.0455006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:54.0455533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:54.0634755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:54.0635515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:54.0636688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:54.0637428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:54.1700270Z dist init r=3, world=4 2022-08-17T12:13:54.1833183Z dist init r=2, world=4 2022-08-17T12:13:54.2061817Z dist init r=1, world=4 2022-08-17T12:13:54.2243082Z dist init r=0, world=4 2022-08-17T12:13:54.2443985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:54.2648343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:54.2750428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:54.2750943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:54.2751673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:54.2752240Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:54.2752791Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:54.2850812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:54.2859407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:54.2860333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:54.2861135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:54.2861928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:54.5999293Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:54.6006229Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-08-17T12:13:54.6054553Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3682 2022-08-17T12:13:54.6093159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3683 2022-08-17T12:13:54.6124216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3684 2022-08-17T12:13:54.6154647Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3685 2022-08-17T12:13:55.6778395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:55.6779002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:55.6780208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:55.6780857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:55.7292896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:55.7293643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:55.7295017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:55.7295652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:55.7523416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:55.7524050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:55.7524878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:55.7525528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:55.7553044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:55.7553665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:55.7555758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:55.7556353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:55.8371656Z dist init r=1, world=4 2022-08-17T12:13:55.8882218Z dist init r=3, world=4 2022-08-17T12:13:55.9109495Z dist init r=0, world=4 2022-08-17T12:13:55.9142963Z dist init r=2, world=4 2022-08-17T12:13:55.9488012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:55.9588010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:55.9691003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:55.9691756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:55.9693018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:55.9693581Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:55.9694284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:55.9695119Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:55.9697936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:55.9698618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:55.9699297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:55.9797670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:56.3196479Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:13:56.3202814Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-08-17T12:13:56.3251518Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3738 2022-08-17T12:13:56.3289934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3739 2022-08-17T12:13:56.3321681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3740 2022-08-17T12:13:56.3352595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3741 2022-08-17T12:13:57.3910734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:57.3911493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:57.3912296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:57.3912849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:57.3914137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:57.3914806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:57.3915627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:57.3916238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:57.4260679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:57.4261229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:57.4263215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:57.4263774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:57.4368046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:57.4368615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:57.4370284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:57.4370953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:57.5515931Z dist init r=2, world=4 2022-08-17T12:13:57.5520230Z dist init r=0, world=4 2022-08-17T12:13:57.5887876Z dist init r=3, world=4 2022-08-17T12:13:57.5997200Z dist init r=1, world=4 2022-08-17T12:13:57.6311522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:57.6312412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:57.6312921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:57.6313323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:57.6313957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:57.6314500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:57.6315049Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:57.6315596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:57.6418167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:57.6418757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:57.6419338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:57.6420055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:57.9392300Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:57.9398255Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-08-17T12:13:57.9445519Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3794 2022-08-17T12:13:57.9483077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3795 2022-08-17T12:13:57.9512287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3796 2022-08-17T12:13:57.9541992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3797 2022-08-17T12:13:58.9792781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:58.9793407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:58.9814444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:58.9815123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:58.9920914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:58.9921892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:58.9923549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:58.9924569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:58.9932290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:58.9932910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:58.9936186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:58.9936805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:58.9977907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:13:58.9978553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:13:58.9981501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:13:58.9982153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:13:59.1401971Z dist init r=2, world=4 2022-08-17T12:13:59.1527690Z dist init r=0, world=4 2022-08-17T12:13:59.1529881Z dist init r=1, world=4 2022-08-17T12:13:59.1584965Z dist init r=3, world=4 2022-08-17T12:13:59.1842052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:13:59.1842639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:13:59.1943829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:13:59.1944468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:13:59.1945447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:59.1946110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:59.1947012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:59.1947771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:13:59.1952728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:13:59.1953439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:13:59.1953796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:13:59.1954153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:13:59.5580763Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:13:59.5586913Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-08-17T12:13:59.5636046Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3850 2022-08-17T12:13:59.5674860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3851 2022-08-17T12:13:59.5706844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3852 2022-08-17T12:13:59.5737762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3853 2022-08-17T12:14:00.6015141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:00.6015786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:00.6016959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:00.6017662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:00.6111020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:00.6111800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:00.6114893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:00.6115627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:00.6116438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:00.6117046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:00.6118635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:00.6119280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:00.6372913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:00.6373750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:00.6375399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:00.6375944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:00.7605668Z dist init r=0, world=4 2022-08-17T12:14:00.7714320Z dist init r=2, world=4 2022-08-17T12:14:00.7721563Z dist init r=1, world=4 2022-08-17T12:14:00.7949669Z dist init r=3, world=4 2022-08-17T12:14:00.8126545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:00.8126986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:00.8228532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:00.8229631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:00.8230286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:00.8231268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:00.8231825Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:00.8234195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:00.8236773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:00.8237474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:00.8238038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:00.8238616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:01.1777413Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:01.1783669Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-08-17T12:14:01.1832253Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3906 2022-08-17T12:14:01.1872676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3907 2022-08-17T12:14:01.1904269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3908 2022-08-17T12:14:01.1934891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3909 2022-08-17T12:14:02.2292060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:02.2292708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:02.2293818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:02.2294454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:02.2298825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:02.2299463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:02.2302818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:02.2303492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:02.2449154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:02.2450195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:02.2451275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:02.2451885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:02.2467587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:02.2468216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:02.2470262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:02.2470888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:02.3911607Z dist init r=1, world=4 2022-08-17T12:14:02.3912028Z dist init r=2, world=4 2022-08-17T12:14:02.4050592Z dist init r=3, world=4 2022-08-17T12:14:02.4061624Z dist init r=0, world=4 2022-08-17T12:14:02.4260558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:02.4463899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:02.4567190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:02.4568071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:02.4568537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:02.4569178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:02.4569726Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:02.4570270Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:02.4573606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:02.4574146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:02.4574746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:02.4575304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:02.7974681Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:02.7981547Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-08-17T12:14:02.8032034Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3962 2022-08-17T12:14:02.8071291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3963 2022-08-17T12:14:02.8101749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3964 2022-08-17T12:14:02.8131047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3965 2022-08-17T12:14:03.8838703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:03.8839433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:03.8840448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:03.8841076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:03.9035795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:03.9036572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:03.9037608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:03.9038543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:03.9304119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:03.9304811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:03.9305976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:03.9306628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:03.9416195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:03.9416845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:03.9419073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:03.9419732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:04.0438857Z dist init r=0, world=4 2022-08-17T12:14:04.0647626Z dist init r=1, world=4 2022-08-17T12:14:04.0906317Z dist init r=2, world=4 2022-08-17T12:14:04.0989084Z dist init r=3, world=4 2022-08-17T12:14:04.1116737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:04.1218269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:04.1218996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:04.1219695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:04.1220777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:04.1221775Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:04.1222639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:04.1223189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:04.1326064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:04.1326654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:04.1327233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:04.1327715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:04.4172062Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:04.4172404Z 2022-08-17T12:14:04.4172723Z ---------------------------------------------------------------------- 2022-08-17T12:14:04.4173019Z Ran 16 tests in 26.378s 2022-08-17T12:14:04.4173141Z 2022-08-17T12:14:04.4173220Z OK (skipped=16) 2022-08-17T12:14:04.4173321Z 2022-08-17T12:14:04.4173409Z Generating XML reports... 2022-08-17T12:14:04.4214270Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220817121338.xml 2022-08-17T12:14:04.4224471Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220817121338.xml 2022-08-17T12:14:04.7443078Z Running distributed/fsdp/test_fsdp_comm ... [2022-08-17 12:14:04.743877] 2022-08-17T12:14:04.7443715Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:14:04.743966] 2022-08-17T12:14:05.8844445Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm 2022-08-17T12:14:05.8856752Z 2022-08-17T12:14:05.8857278Z Running tests... 2022-08-17T12:14:05.8857725Z ---------------------------------------------------------------------- 2022-08-17T12:14:05.8876311Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-08-17T12:14:06.1493743Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4029 2022-08-17T12:14:06.1521800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4030 2022-08-17T12:14:06.1550073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4031 2022-08-17T12:14:06.1579717Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4032 2022-08-17T12:14:07.2031992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:07.2032656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:07.2033869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:07.2034522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:07.2207708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:07.2208429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:07.2209658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:07.2210335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:07.2216974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:07.2217645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:07.2220804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:07.2221415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:07.2295110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:07.2295788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:07.2298837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:07.2299439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:07.3620306Z dist init r=0, world=4 2022-08-17T12:14:07.3788336Z dist init r=3, world=4 2022-08-17T12:14:07.3805346Z dist init r=1, world=4 2022-08-17T12:14:07.3889165Z dist init r=2, world=4 2022-08-17T12:14:07.4101183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:07.4201693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:07.4304052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:07.4304753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:07.4305706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:07.4306376Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:07.4307022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:07.4307558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:07.4310759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:07.4311844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:07.4312524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:07.4312934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:07.7624403Z skip: Need at least 2 CUDA devices (1.876s) 2022-08-17T12:14:07.7643767Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-08-17T12:14:07.7692552Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4085 2022-08-17T12:14:07.7729989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4086 2022-08-17T12:14:07.7759389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4087 2022-08-17T12:14:07.7789077Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4088 2022-08-17T12:14:08.7925432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:08.7926506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:08.7927393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:08.7928010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:08.8110679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:08.8111411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:08.8112431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:08.8113079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:08.8201511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:08.8202206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:08.8203669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:08.8224055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:08.8258204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:08.8258867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:08.8261123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:08.8261770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:08.9551552Z dist init r=1, world=4 2022-08-17T12:14:08.9725033Z dist init r=2, world=4 2022-08-17T12:14:08.9876459Z dist init r=0, world=4 2022-08-17T12:14:09.0013747Z dist init r=3, world=4 2022-08-17T12:14:09.0291075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:09.0391387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:09.0492369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:09.0493092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:09.0494134Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:09.0494855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:09.0495777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:09.0496473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:09.0600384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:09.0601348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:09.0601937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:09.0602552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:09.3828289Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:09.3847065Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-08-17T12:14:09.3894694Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4141 2022-08-17T12:14:09.3932170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4142 2022-08-17T12:14:09.3960637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4143 2022-08-17T12:14:09.3990031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4144 2022-08-17T12:14:10.3744825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:10.3745423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:10.3746688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:10.3747313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:10.3769884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:10.3770548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:10.3773673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:10.3774321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:10.3954322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:10.3954960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:10.3956433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:10.3957008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:10.3958408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:10.3959009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:10.3962324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:10.3963029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:10.5331298Z dist init r=3, world=4 2022-08-17T12:14:10.5357137Z dist init r=2, world=4 2022-08-17T12:14:10.5534550Z dist init r=1, world=4 2022-08-17T12:14:10.5536225Z dist init r=0, world=4 2022-08-17T12:14:10.5949148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:10.5949556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:10.6050413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:10.6051387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:10.6053006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:10.6053560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:10.6054111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:10.6054659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:10.6157249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:10.6157642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:10.6158086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:10.6158447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:10.9028438Z skip: Need at least 2 CUDA devices (1.520s) 2022-08-17T12:14:10.9049515Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-08-17T12:14:10.9097915Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4197 2022-08-17T12:14:10.9133559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4198 2022-08-17T12:14:10.9161876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4199 2022-08-17T12:14:10.9189858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4200 2022-08-17T12:14:11.9653748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:11.9654407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:11.9655247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:11.9655890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:12.0107182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:12.0107792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:12.0108602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:12.0109220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:12.0145385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:12.0146054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:12.0147548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:12.0148187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:12.0669385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:12.0670059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:12.0671647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:12.0672298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:12.1284644Z dist init r=2, world=4 2022-08-17T12:14:12.1747778Z dist init r=0, world=4 2022-08-17T12:14:12.1870473Z dist init r=1, world=4 2022-08-17T12:14:12.2257986Z dist init r=3, world=4 2022-08-17T12:14:12.2368783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:12.2401967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:12.2504693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:12.2505453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:12.2506093Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:12.2506647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:12.2507194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:12.2572571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:12.2612516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:12.2613346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:12.2613941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:12.2614487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:12.6231216Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:14:12.6249977Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-08-17T12:14:12.6298548Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4253 2022-08-17T12:14:12.6337014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4254 2022-08-17T12:14:12.6366704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4255 2022-08-17T12:14:12.6395949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4256 2022-08-17T12:14:13.6592260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:13.6592897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:13.6594295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:13.6594947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:13.6749278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:13.6749929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:13.6751733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:13.6752361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:13.6759813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:13.6760444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:13.6762784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:13.6763423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:13.6793575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:13.6794185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:13.6796537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:13.6797164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:13.8190627Z dist init r=2, world=4 2022-08-17T12:14:13.8351794Z dist init r=3, world=4 2022-08-17T12:14:13.8371577Z dist init r=0, world=4 2022-08-17T12:14:13.8378847Z dist init r=1, world=4 2022-08-17T12:14:13.8692611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:13.8693240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:13.8793717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:13.8794459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:13.8795199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:13.8796069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:13.8796934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:13.8797492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:13.8901100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:13.8901709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:13.8902295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:13.8902861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:14.2438021Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:14:14.2456977Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-08-17T12:14:14.2502924Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4309 2022-08-17T12:14:14.2537792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4310 2022-08-17T12:14:14.2567296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4311 2022-08-17T12:14:14.2596295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4312 2022-08-17T12:14:15.2817289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:15.2818029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:15.2818887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:15.2819544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:15.2969958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:15.2970650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:15.2972069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:15.2973235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:15.3131118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:15.3131701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:15.3132391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:15.3133174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:15.3297454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:15.3298148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:15.3300460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:15.3301106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:15.4432138Z dist init r=2, world=4 2022-08-17T12:14:15.4592020Z dist init r=1, world=4 2022-08-17T12:14:15.4732480Z dist init r=0, world=4 2022-08-17T12:14:15.4880199Z dist init r=3, world=4 2022-08-17T12:14:15.4990640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:15.5106236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:15.5106826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:15.5107312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:15.5108112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:15.5108673Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:15.5109209Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:15.5193827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:15.5213547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:15.5214162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:15.5214687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:15.5215228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:15.8637625Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:15.8656620Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-08-17T12:14:15.8705378Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4365 2022-08-17T12:14:15.8742507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4366 2022-08-17T12:14:15.8773065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4367 2022-08-17T12:14:15.8802758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4368 2022-08-17T12:14:16.8938111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:16.8938742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:16.8939647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:16.8940313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:16.9234536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:16.9235219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:16.9236088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:16.9236749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:17.0049580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:17.0050282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:17.0051695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:17.0052402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:17.0288399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:17.0289109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:17.0290468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:17.0291181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:17.0513099Z dist init r=0, world=4 2022-08-17T12:14:17.0819277Z dist init r=1, world=4 2022-08-17T12:14:17.1628487Z dist init r=2, world=4 2022-08-17T12:14:17.1834751Z dist init r=3, world=4 2022-08-17T12:14:17.1945579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:17.2037693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:17.2038409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:17.2039042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:17.2039687Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:17.2040248Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:17.2040813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:17.2048032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:17.2146894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:17.2147486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:17.2148057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:17.2148637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:17.5843633Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:14:17.5862512Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-08-17T12:14:17.5911257Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4421 2022-08-17T12:14:17.5949904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4422 2022-08-17T12:14:17.5980290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4423 2022-08-17T12:14:17.6010373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4424 2022-08-17T12:14:18.5971066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:18.5971762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:18.5972941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:18.5973615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:18.6117056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:18.6117728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:18.6119344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:18.6119927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:18.6296454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:18.6297262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:18.6298124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:18.6298768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:18.6582891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:18.6583579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:18.6585033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:18.6585879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:18.7572046Z dist init r=3, world=4 2022-08-17T12:14:18.7708950Z dist init r=0, world=4 2022-08-17T12:14:18.7895914Z dist init r=2, world=4 2022-08-17T12:14:18.8153515Z dist init r=1, world=4 2022-08-17T12:14:18.8468650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:18.8469430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:18.8469949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:18.8470765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:18.8471288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:18.8471827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:18.8472363Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:18.8472911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:18.8575894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:18.8576618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:18.8577271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:18.8577809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:19.2050780Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:14:19.2051037Z 2022-08-17T12:14:19.2051892Z ---------------------------------------------------------------------- 2022-08-17T12:14:19.2052197Z Ran 8 tests in 13.319s 2022-08-17T12:14:19.2052330Z 2022-08-17T12:14:19.2052411Z OK (skipped=8) 2022-08-17T12:14:19.2052609Z 2022-08-17T12:14:19.2052688Z Generating XML reports... 2022-08-17T12:14:19.2096052Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220817121405.xml 2022-08-17T12:14:19.5253107Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-08-17 12:14:19.524894] 2022-08-17T12:14:19.5253785Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:14:19.524980] 2022-08-17T12:14:20.6553934Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks 2022-08-17T12:14:20.6566156Z 2022-08-17T12:14:20.6566298Z Running tests... 2022-08-17T12:14:20.6566891Z ---------------------------------------------------------------------- 2022-08-17T12:14:20.6574604Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:14:20.6581390Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:14:20.6593237Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:20.9247563Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4488 2022-08-17T12:14:20.9275682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4489 2022-08-17T12:14:20.9303268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4490 2022-08-17T12:14:20.9334625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4491 2022-08-17T12:14:22.0100329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:22.0100990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:22.0103748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:22.0104506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:22.0253423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:22.0254075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:22.0256581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:22.0257231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:22.0561524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:22.0562163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:22.0565376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:22.0566128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:22.0884690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:22.0885346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:22.0888636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:22.0889232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:22.1715878Z dist init r=3, world=4 2022-08-17T12:14:22.1854677Z dist init r=0, world=4 2022-08-17T12:14:22.2149350Z dist init r=1, world=4 2022-08-17T12:14:22.2463149Z dist init r=2, world=4 2022-08-17T12:14:22.2769288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:22.2869697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:22.2972615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:22.2973850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:22.2974450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:22.2974985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:22.2975532Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:22.2976085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:22.3080146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:22.3080687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:22.3081157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:22.3081698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:22.6381286Z skip: Need at least 2 CUDA devices (1.980s) 2022-08-17T12:14:22.6396649Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:22.6444781Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4544 2022-08-17T12:14:22.6481738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4545 2022-08-17T12:14:22.6510788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4546 2022-08-17T12:14:22.6539972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4547 2022-08-17T12:14:23.6636758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:23.6637399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:23.6639612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:23.6640269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:23.7030821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:23.7031555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:23.7034346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:23.7034986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:23.7040715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:23.7041361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:23.7045225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:23.7046036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:23.7049111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:23.7049778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:23.7053786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:23.7054464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:23.8247009Z dist init r=1, world=4 2022-08-17T12:14:23.8640921Z dist init r=3, world=4 2022-08-17T12:14:23.8643090Z dist init r=2, world=4 2022-08-17T12:14:23.8656038Z dist init r=0, world=4 2022-08-17T12:14:23.8857767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:23.9060769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:23.9164843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:23.9165345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:23.9166152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:23.9166717Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:23.9167323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:23.9167919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:23.9170053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:23.9170563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:23.9170947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:23.9171470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:24.2581033Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:24.2596129Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:24.2647437Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4600 2022-08-17T12:14:24.2686511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4601 2022-08-17T12:14:24.2715750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4602 2022-08-17T12:14:24.2746860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4603 2022-08-17T12:14:25.2544370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:25.2545052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:25.2547538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:25.2548196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:25.2793317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:25.2793973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:25.2796794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:25.2797445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:25.3029524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:25.3030204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:25.3032893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:25.3033525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:25.3129607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:25.3130305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:25.3134490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:25.3135129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:25.4129757Z dist init r=3, world=4 2022-08-17T12:14:25.4386140Z dist init r=2, world=4 2022-08-17T12:14:25.4621877Z dist init r=1, world=4 2022-08-17T12:14:25.4706177Z dist init r=0, world=4 2022-08-17T12:14:25.4897223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:25.5033799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:25.5034286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:25.5034678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:25.5035325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:25.5035882Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:25.5036429Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:25.5099944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:25.5141844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:25.5143260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:25.5144638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:25.5145054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:25.8786302Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:25.8801457Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-08-17T12:14:25.8848974Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4656 2022-08-17T12:14:25.8887317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4657 2022-08-17T12:14:25.8917891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4658 2022-08-17T12:14:25.8947370Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4659 2022-08-17T12:14:26.9173065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:26.9173681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:26.9176582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:26.9177523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:26.9210444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:26.9211113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:26.9214999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:26.9215590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:26.9277588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:26.9278223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:26.9282126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:26.9282770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:26.9461158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:26.9461709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:26.9464346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:26.9464907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:27.0801826Z dist init r=1, world=4 2022-08-17T12:14:27.0817473Z dist init r=2, world=4 2022-08-17T12:14:27.0887353Z dist init r=3, world=4 2022-08-17T12:14:27.1052391Z dist init r=0, world=4 2022-08-17T12:14:27.1464777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:27.1566276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:27.1667936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:27.1669381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:27.1670255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:27.1671511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:27.1672162Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:27.1672714Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:27.1676363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:27.1677154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:27.1677838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:27.1679977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:27.4987401Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:27.5002203Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:27.5050626Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4712 2022-08-17T12:14:27.5088198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4713 2022-08-17T12:14:27.5117686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4714 2022-08-17T12:14:27.5148695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4715 2022-08-17T12:14:28.5194825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:28.5195482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:28.5199501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:28.5200160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:28.5200981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:28.5201596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:28.5203007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:28.5203616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:28.5899343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:28.5900274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:28.5901679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:28.5902329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:28.6010116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:28.6010787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:28.6013301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:28.6013939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:28.6796544Z dist init r=0, world=4 2022-08-17T12:14:28.6796916Z dist init r=2, world=4 2022-08-17T12:14:28.7452303Z dist init r=1, world=4 2022-08-17T12:14:28.7572285Z dist init r=3, world=4 2022-08-17T12:14:28.7763627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:28.7764049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:28.7865577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:28.7866203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:28.7867066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:28.7867855Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:28.7868453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:28.7869019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:28.7973884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:28.7974528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:28.7975091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:28.7975460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:29.1187852Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:29.1203226Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:29.1251008Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4768 2022-08-17T12:14:29.1287512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4769 2022-08-17T12:14:29.1315748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4770 2022-08-17T12:14:29.1345426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4771 2022-08-17T12:14:30.1129122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:30.1129746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:30.1132703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:30.1133399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:30.1497543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:30.1498159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:30.1501167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:30.1501814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:30.1652958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:30.1653630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:30.1656807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:30.1657413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:30.1754963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:30.1755606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:30.1758669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:30.1759338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:30.2721693Z dist init r=0, world=4 2022-08-17T12:14:30.3099460Z dist init r=1, world=4 2022-08-17T12:14:30.3261983Z dist init r=3, world=4 2022-08-17T12:14:30.3331709Z dist init r=2, world=4 2022-08-17T12:14:30.3470730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:30.3612813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:30.3613942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:30.3614488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:30.3614892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:30.3615429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:30.3615984Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:30.3674135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:30.3721716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:30.3722323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:30.3722932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:30.3723374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:30.7385038Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:30.7400110Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-08-17T12:14:30.7449074Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4824 2022-08-17T12:14:30.7487033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4825 2022-08-17T12:14:30.7516838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4826 2022-08-17T12:14:30.7547181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4827 2022-08-17T12:14:31.8012172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:31.8012810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:31.8015336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:31.8016029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:31.8336838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:31.8337523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:31.8340657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:31.8341233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:31.8397613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:31.8398268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:31.8402198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:31.8402868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:31.8627137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:31.8628006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:31.8630724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:31.8631334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:31.9584993Z dist init r=1, world=4 2022-08-17T12:14:31.9939215Z dist init r=0, world=4 2022-08-17T12:14:31.9966992Z dist init r=3, world=4 2022-08-17T12:14:32.0184076Z dist init r=2, world=4 2022-08-17T12:14:32.0378310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:32.0553183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:32.0654268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:32.0654819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:32.0655735Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:32.0656298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:32.0656855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:32.0682014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:32.0762654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:32.0763485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:32.0764176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:32.0764762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:32.3586729Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:32.3594113Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:14:32.3599476Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:14:32.3605216Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:32.3653830Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4880 2022-08-17T12:14:32.3691548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4881 2022-08-17T12:14:32.3720502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4882 2022-08-17T12:14:32.3750313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4883 2022-08-17T12:14:33.4207802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:33.4208486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:33.4210900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:33.4211505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:33.4290104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:33.4290753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:33.4294109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:33.4294748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:33.4329396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:33.4330049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:33.4333982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:33.4334642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:33.4469360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:33.4469953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:33.4473406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:33.4473969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:33.5802937Z dist init r=0, world=4 2022-08-17T12:14:33.5885362Z dist init r=1, world=4 2022-08-17T12:14:33.5931139Z dist init r=2, world=4 2022-08-17T12:14:33.6055075Z dist init r=3, world=4 2022-08-17T12:14:33.6214464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:33.6214889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:33.6316313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:33.6316997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:33.6317868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:33.6318702Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:33.6319357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:33.6319893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:33.6323847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:33.6324673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:33.6325542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:33.6326923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:33.9790025Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:33.9801634Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-08-17T12:14:33.9848758Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4936 2022-08-17T12:14:33.9886880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4937 2022-08-17T12:14:33.9916440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4938 2022-08-17T12:14:33.9946330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4939 2022-08-17T12:14:34.9939240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:34.9939924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:34.9942495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:34.9943428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:35.0254334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:35.0254965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:35.0256687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:35.0257356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:35.0381713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:35.0382382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:35.0385558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:35.0386224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:35.0523399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:35.0524052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:35.0527172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:35.0527849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:35.1496900Z dist init r=2, world=4 2022-08-17T12:14:35.1830333Z dist init r=3, world=4 2022-08-17T12:14:35.2104054Z dist init r=1, world=4 2022-08-17T12:14:35.2133344Z dist init r=0, world=4 2022-08-17T12:14:35.2340868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:35.2513209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:35.2614695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:35.2615390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:35.2616060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:35.2616605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:35.2617155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:35.2644523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:35.2721924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:35.2722479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:35.2723040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:35.2723599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:35.5985039Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:35.5985342Z 2022-08-17T12:14:35.5985807Z ---------------------------------------------------------------------- 2022-08-17T12:14:35.5986078Z Ran 13 tests in 14.942s 2022-08-17T12:14:35.5986198Z 2022-08-17T12:14:35.5986274Z OK (skipped=13) 2022-08-17T12:14:35.5986388Z 2022-08-17T12:14:35.5986478Z Generating XML reports... 2022-08-17T12:14:35.6038021Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220817121420.xml 2022-08-17T12:14:35.9262394Z Running distributed/fsdp/test_fsdp_core ... [2022-08-17 12:14:35.925826] 2022-08-17T12:14:35.9263227Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:14:35.925917] 2022-08-17T12:14:37.0665073Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-08-17T12:14:37.0683279Z 2022-08-17T12:14:37.0683439Z Running tests... 2022-08-17T12:14:37.0683840Z ---------------------------------------------------------------------- 2022-08-17T12:14:37.0690759Z test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks) 2022-08-17T12:14:37.3356996Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5003 2022-08-17T12:14:37.3383508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5004 2022-08-17T12:14:37.3411142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5005 2022-08-17T12:14:37.3443107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5006 2022-08-17T12:14:38.4554399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:38.4555042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:38.4555866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:38.4556460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:38.4557243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:38.4558055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:38.4559179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:38.4559843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:38.4901228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:38.4901883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:38.4902678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:38.4903247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:38.4981485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:38.4982134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:38.4984028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:38.4984964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:38.6153980Z dist init r=2, world=4 2022-08-17T12:14:38.6159567Z dist init r=0, world=4 2022-08-17T12:14:38.6501298Z dist init r=3, world=4 2022-08-17T12:14:38.6587147Z dist init r=1, world=4 2022-08-17T12:14:38.6899861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:38.6900585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:38.6901318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:38.6901827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:38.6902448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:38.6903024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:38.6903749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:38.6904299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:38.7006942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:38.7007540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:38.7008111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:38.7008701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:39.0492121Z skip: Need at least 2 CUDA devices (1.981s) 2022-08-17T12:14:39.0498008Z test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks) 2022-08-17T12:14:39.0547345Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5059 2022-08-17T12:14:39.0585206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5060 2022-08-17T12:14:39.0615785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5061 2022-08-17T12:14:39.0647096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5062 2022-08-17T12:14:40.0953545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:40.0954234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:40.0955474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:40.0956118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:40.1004515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:40.1005192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:40.1007821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:40.1008507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:40.1117860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:40.1118641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:40.1119953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:40.1120829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:40.1226952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:40.1227556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:40.1229716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:40.1230303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:40.2565186Z dist init r=1, world=4 2022-08-17T12:14:40.2637693Z dist init r=0, world=4 2022-08-17T12:14:40.2720569Z dist init r=2, world=4 2022-08-17T12:14:40.2815100Z dist init r=3, world=4 2022-08-17T12:14:40.2932749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:40.3032793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:40.3134648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:40.3135342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:40.3136566Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:40.3137136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:40.3137681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:40.3138229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:40.3243472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:40.3244143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:40.3244714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:40.3245263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:40.6687897Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:40.6693634Z test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks) 2022-08-17T12:14:40.6742696Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5115 2022-08-17T12:14:40.6779563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5116 2022-08-17T12:14:40.6808650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5117 2022-08-17T12:14:40.6838956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5118 2022-08-17T12:14:41.6767156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:41.6767832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:41.6771378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:41.6772014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:41.6773565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:41.6774159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:41.6777742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:41.6778416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:41.7137671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:41.7138555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:41.7139240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:41.7139772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:41.7802374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:41.7803089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:41.7804544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:41.7805219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:41.8376308Z dist init r=3, world=4 2022-08-17T12:14:41.8394338Z dist init r=2, world=4 2022-08-17T12:14:41.8745865Z dist init r=0, world=4 2022-08-17T12:14:41.9368471Z dist init r=1, world=4 2022-08-17T12:14:41.9491114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:41.9592241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:41.9695177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:41.9695922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:41.9696655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:41.9697279Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:41.9697829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:41.9698551Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:41.9701341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:41.9702024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:41.9703012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:41.9705184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:42.2879242Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:42.2889713Z test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks) 2022-08-17T12:14:42.2938522Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5171 2022-08-17T12:14:42.2977542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5172 2022-08-17T12:14:42.3009923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5173 2022-08-17T12:14:42.3050694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5174 2022-08-17T12:14:43.3581352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:43.3582306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:43.3583305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:43.3583954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:43.3727817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:43.3728455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:43.3729583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:43.3730173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:43.3979645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:43.3980193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:43.3982233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:43.3982717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:43.4175706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:43.4176495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:43.4177740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:43.4178479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:43.5214509Z dist init r=0, world=4 2022-08-17T12:14:43.5363720Z dist init r=3, world=4 2022-08-17T12:14:43.5770633Z dist init r=1, world=4 2022-08-17T12:14:43.5781187Z dist init r=2, world=4 2022-08-17T12:14:43.6129901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:43.6230087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:43.6332771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:43.6333467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:43.6334412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:43.6335060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:43.6335742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:43.6336294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:43.6440332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:43.6441059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:43.6441635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:43.6442344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:44.0093673Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:14:44.0103069Z test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks) 2022-08-17T12:14:44.0152207Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5227 2022-08-17T12:14:44.0190582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5228 2022-08-17T12:14:44.0220627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5229 2022-08-17T12:14:44.0249943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5230 2022-08-17T12:14:45.0497342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:45.0497985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:45.0499200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:45.0500147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:45.0611804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:45.0612472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:45.0613710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:45.0614291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:45.0719699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:45.0720364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:45.0721725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:45.0722364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:45.0848009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:45.0848626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:45.0850873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:45.0851446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:45.2100513Z dist init r=3, world=4 2022-08-17T12:14:45.2207763Z dist init r=1, world=4 2022-08-17T12:14:45.2332588Z dist init r=2, world=4 2022-08-17T12:14:45.2443491Z dist init r=0, world=4 2022-08-17T12:14:45.2836570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:45.2837028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:45.2922692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:45.2923679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:45.2924732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:45.2925771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:45.2926842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:45.2927493Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:45.2932255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:45.2933132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:45.2933954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:45.3009902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:45.6290640Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:45.6301870Z test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks) 2022-08-17T12:14:45.6377683Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5283 2022-08-17T12:14:45.6417594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5284 2022-08-17T12:14:45.6447131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5285 2022-08-17T12:14:45.6485150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5286 2022-08-17T12:14:46.6208193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:46.6208863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:46.6210223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:46.6210876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:46.6211711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:46.6212331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:46.6215576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:46.6216215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:46.6269695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:46.6270360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:46.6273200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:46.6273786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:46.6286547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:46.6287202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:46.6289152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:46.6289797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:46.7832007Z dist init r=2, world=4 2022-08-17T12:14:46.7888894Z dist init r=3, world=4 2022-08-17T12:14:46.7895362Z dist init r=1, world=4 2022-08-17T12:14:46.7947802Z dist init r=0, world=4 2022-08-17T12:14:46.8359308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:46.8461584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:46.8462729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:46.8463397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:46.8463997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:46.8464908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:46.8465656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:46.8466227Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:46.8470447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:46.8471392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:46.8471976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:46.8473425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:47.1522731Z skip: Need at least 2 CUDA devices (1.523s) 2022-08-17T12:14:47.1533556Z test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks) 2022-08-17T12:14:47.1581710Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5339 2022-08-17T12:14:47.1617916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5340 2022-08-17T12:14:47.1646838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5341 2022-08-17T12:14:47.1675835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5342 2022-08-17T12:14:48.1650416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:48.1651081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:48.1654304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:48.1654966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:48.1979746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:48.1980437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:48.1981295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:48.1981967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:48.2020723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:48.2021565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:48.2023485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:48.2024145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:48.2099959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:48.2100564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:48.2103469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:48.2104167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:48.3253759Z dist init r=1, world=4 2022-08-17T12:14:48.3591009Z dist init r=0, world=4 2022-08-17T12:14:48.3629391Z dist init r=3, world=4 2022-08-17T12:14:48.3679410Z dist init r=2, world=4 2022-08-17T12:14:48.3840421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:48.3967305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:48.3967908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:48.3968318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:48.3969036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:48.3969593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:48.3970146Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:48.4043675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:48.4075370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:48.4075868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:48.4076433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:48.4076984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:48.7715827Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:48.7729056Z test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) 2022-08-17T12:14:48.7777653Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5395 2022-08-17T12:14:48.7818124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5396 2022-08-17T12:14:48.7847004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5397 2022-08-17T12:14:48.7886027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5398 2022-08-17T12:14:49.8286994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:49.8287634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:49.8291093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:49.8291697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:49.8447168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:49.8448279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:49.8449499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:49.8450188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:49.8715529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:49.8716875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:49.8718040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:49.8718698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:49.8850224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:49.8850865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:49.8852634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:49.8853242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:49.9934631Z dist init r=2, world=4 2022-08-17T12:14:50.0061118Z dist init r=3, world=4 2022-08-17T12:14:50.0309742Z dist init r=1, world=4 2022-08-17T12:14:50.0427795Z dist init r=0, world=4 2022-08-17T12:14:50.0647775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:50.0748599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:50.0749314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:50.0750015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:50.0751149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:50.0752147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:50.0752680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:50.0753231Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:50.0855939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:50.0856750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:50.0857838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:50.0858479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:50.3924949Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:14:50.3935861Z test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) 2022-08-17T12:14:50.3984477Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5451 2022-08-17T12:14:50.4023657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5452 2022-08-17T12:14:50.4055211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5453 2022-08-17T12:14:50.4086015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5454 2022-08-17T12:14:51.4412642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:51.4413298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:51.4414512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:51.4415144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:51.4461042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:51.4461977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:51.4463966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:51.4464595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:51.4706854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:51.4707542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:51.4709195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:51.4709820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:51.4966682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:51.4967318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:51.4968178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:51.4968761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:51.6033070Z dist init r=1, world=4 2022-08-17T12:14:51.6063031Z dist init r=3, world=4 2022-08-17T12:14:51.6317137Z dist init r=0, world=4 2022-08-17T12:14:51.6567936Z dist init r=2, world=4 2022-08-17T12:14:51.6847126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:51.6847577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:51.6948681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:51.6949311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:51.6950285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:51.6951239Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:51.6951905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:51.6952652Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:51.6956820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:51.6957421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:51.6957977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:51.6959182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:52.0125075Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:52.0136039Z test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) 2022-08-17T12:14:52.0184325Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5507 2022-08-17T12:14:52.0221959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5508 2022-08-17T12:14:52.0251231Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5509 2022-08-17T12:14:52.0289629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5510 2022-08-17T12:14:53.0369650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:53.0370424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:53.0371254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:53.0371885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:53.0990966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:53.0991727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:53.0992581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:53.0993242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:53.1055363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:53.1055956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:53.1058041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:53.1058621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:53.1173748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:53.1174419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:53.1175969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:53.1176607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:53.1963518Z dist init r=2, world=4 2022-08-17T12:14:53.2592806Z dist init r=1, world=4 2022-08-17T12:14:53.2663446Z dist init r=0, world=4 2022-08-17T12:14:53.2772277Z dist init r=3, world=4 2022-08-17T12:14:53.3005817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:53.3006533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:53.3108219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:53.3108869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:53.3109988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:53.3110769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:53.3111685Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:53.3112511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:53.3116293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:53.3116875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:53.3117440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:53.3117992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:53.6330287Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:53.6340289Z test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) 2022-08-17T12:14:53.6389264Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5563 2022-08-17T12:14:53.6425968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5564 2022-08-17T12:14:53.6455977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5565 2022-08-17T12:14:53.6487869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5566 2022-08-17T12:14:54.6625517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:54.6626256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:54.6627767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:54.6628423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:54.6680046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:54.6680667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:54.6683077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:54.6683672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:54.6834144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:54.6834823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:54.6838728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:54.6839352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:54.6840128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:54.6840797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:54.6841655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:54.6842261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:54.8249544Z dist init r=3, world=4 2022-08-17T12:14:54.8278696Z dist init r=1, world=4 2022-08-17T12:14:54.8438570Z dist init r=2, world=4 2022-08-17T12:14:54.8448284Z dist init r=0, world=4 2022-08-17T12:14:54.8750434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:54.8851815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:54.8852513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:54.8853226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:54.8854005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:54.8854659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:54.8855208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:54.8855758Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:54.8959162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:54.8959847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:54.8960282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:54.8960852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:55.2526696Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:14:55.2534591Z test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP) 2022-08-17T12:14:55.2583812Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5619 2022-08-17T12:14:55.2622046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5620 2022-08-17T12:14:55.2652730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5621 2022-08-17T12:14:55.2682545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5622 2022-08-17T12:14:56.3388940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:56.3389605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:56.3390483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:56.3391218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:56.3528508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:56.3529159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:56.3530457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:56.3531163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:56.3878758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:56.3879529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:56.3880296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:56.3880945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:56.3973672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:56.3974352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:56.3975525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:56.3976180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:56.4990492Z dist init r=0, world=4 2022-08-17T12:14:56.5145473Z dist init r=1, world=4 2022-08-17T12:14:56.5481861Z dist init r=2, world=4 2022-08-17T12:14:56.5580603Z dist init r=3, world=4 2022-08-17T12:14:56.5692437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:56.5704877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:56.5705304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:56.5705739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:56.5706385Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:56.5706948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:56.5707493Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:56.5795255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:56.5813168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:56.5813815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:56.5814176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:56.5814539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:56.8723210Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:56.8730156Z test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP) 2022-08-17T12:14:56.8779904Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5675 2022-08-17T12:14:56.8818019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5676 2022-08-17T12:14:56.8848038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5677 2022-08-17T12:14:56.8878516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5678 2022-08-17T12:14:57.8803728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:57.8804387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:57.8805450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:57.8806291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:57.8890377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:57.8891001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:57.8893166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:57.8893830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:57.9240020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:57.9240583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:57.9241911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:57.9242450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:57.9389256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:57.9390203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:57.9391431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:57.9392102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:58.0408119Z dist init r=2, world=4 2022-08-17T12:14:58.0481938Z dist init r=0, world=4 2022-08-17T12:14:58.0818304Z dist init r=3, world=4 2022-08-17T12:14:58.0983891Z dist init r=1, world=4 2022-08-17T12:14:58.1128518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:58.1228731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:58.1296623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:58.1297379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:58.1298159Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:58.1298721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:58.1330847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:58.1331950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:58.1437357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:58.1437972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:58.1438522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:14:58.1439146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:58.4918695Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:14:58.4925594Z test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-08-17T12:14:58.4973594Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5731 2022-08-17T12:14:58.5011066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5732 2022-08-17T12:14:58.5041022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5733 2022-08-17T12:14:58.5070709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5734 2022-08-17T12:14:59.5064781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:59.5065421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:59.5066575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:59.5067214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:59.5085030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:59.5085658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:59.5088774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:59.5089375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:59.5406871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:59.5407500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:59.5409037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:59.5410097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:59.5649868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:14:59.5650589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:14:59.5651481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:14:59.5652184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:14:59.6672684Z dist init r=2, world=4 2022-08-17T12:14:59.6708324Z dist init r=1, world=4 2022-08-17T12:14:59.7007226Z dist init r=0, world=4 2022-08-17T12:14:59.7241441Z dist init r=3, world=4 2022-08-17T12:14:59.7349681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:14:59.7451367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:14:59.7554110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:14:59.7554955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:14:59.7555768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:59.7556493Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:59.7557329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:59.7558204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:14:59.7560738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:14:59.7561284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:14:59.7561754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:14:59.7562257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:00.1112932Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:00.1119237Z test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP) 2022-08-17T12:15:00.1123570Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:15:00.1128206Z test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP) 2022-08-17T12:15:00.1174420Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5787 2022-08-17T12:15:00.1213460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5788 2022-08-17T12:15:00.1243634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5789 2022-08-17T12:15:00.1273715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5790 2022-08-17T12:15:01.1823032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:01.1823687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:01.1825094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:01.1826041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:01.2032154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:01.2032970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:01.2033811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:01.2034390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:01.2075100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:01.2075737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:01.2077379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:01.2077978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:01.2096024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:01.2096676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:01.2100098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:01.2100771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:01.3459648Z dist init r=2, world=4 2022-08-17T12:15:01.3662824Z dist init r=0, world=4 2022-08-17T12:15:01.3688643Z dist init r=3, world=4 2022-08-17T12:15:01.3714065Z dist init r=1, world=4 2022-08-17T12:15:01.4075398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:01.4177899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:01.4178533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:01.4179137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:01.4179861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:01.4180410Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:01.4180963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:01.4181512Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:01.4285980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:01.4286585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:01.4287169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:01.4287722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:01.7314573Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:01.7321396Z test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-08-17T12:15:01.7372148Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5843 2022-08-17T12:15:01.7412033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5844 2022-08-17T12:15:01.7442901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5845 2022-08-17T12:15:01.7473929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5846 2022-08-17T12:15:02.7595533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:02.7596557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:02.7597402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:02.7597997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:02.7875481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:02.7876448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:02.7877319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:02.7877974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:02.7880375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:02.7881013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:02.7884780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:02.7885643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:02.8026856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:02.8027542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:02.8029015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:02.8029643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:02.9208904Z dist init r=0, world=4 2022-08-17T12:15:02.9484683Z dist init r=3, world=4 2022-08-17T12:15:02.9505338Z dist init r=1, world=4 2022-08-17T12:15:02.9628012Z dist init r=2, world=4 2022-08-17T12:15:02.9795001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:02.9894959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:02.9922375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:02.9922834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:02.9923484Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:02.9924043Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:02.9997689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:02.9998261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:03.0104784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:03.0105491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:03.0105880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:03.0106237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:03.3513710Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:03.3519538Z test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP) 2022-08-17T12:15:03.3567069Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5899 2022-08-17T12:15:03.3605516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5900 2022-08-17T12:15:03.3634930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5901 2022-08-17T12:15:03.3664949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5902 2022-08-17T12:15:04.4422649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:04.4423274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:04.4424067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:04.4424672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:04.4573134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:04.4573786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:04.4574915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:04.4575558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:04.4637522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:04.4638220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:04.4639502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:04.4640196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:04.4697570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:04.4698081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:04.4700849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:04.4701401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:04.6038850Z dist init r=1, world=4 2022-08-17T12:15:04.6186343Z dist init r=3, world=4 2022-08-17T12:15:04.6251926Z dist init r=2, world=4 2022-08-17T12:15:04.6304187Z dist init r=0, world=4 2022-08-17T12:15:04.6565108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:04.6665350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:04.6666867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:04.6667628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:04.6668310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:04.6668885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:04.6669434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:04.6669984Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:04.6773689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:04.6774404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:04.6774973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:04.6775548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:04.9704936Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:04.9712001Z test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP) 2022-08-17T12:15:04.9716517Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:15:04.9721420Z test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-08-17T12:15:04.9724208Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:15:04.9729103Z test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP) 2022-08-17T12:15:04.9776564Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5955 2022-08-17T12:15:04.9814945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5956 2022-08-17T12:15:04.9844576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5957 2022-08-17T12:15:04.9874226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5958 2022-08-17T12:15:05.9904848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:05.9906133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:05.9907338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:05.9907989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:06.0068455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:06.0069082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:06.0070322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:06.0070910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:06.0213533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:06.0214133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:06.0215231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:06.0215911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:06.0450204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:06.0450948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:06.0451935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:06.0452612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:06.1518879Z dist init r=1, world=4 2022-08-17T12:15:06.1675578Z dist init r=2, world=4 2022-08-17T12:15:06.1810062Z dist init r=3, world=4 2022-08-17T12:15:06.2042152Z dist init r=0, world=4 2022-08-17T12:15:06.2220927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:06.2321656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:06.2390198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:06.2390603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:06.2391336Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:06.2391907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:06.2423325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:06.2424029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:06.2497996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:06.2498717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:06.2499327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:06.2499873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:06.5914683Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:06.5920962Z test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP) 2022-08-17T12:15:06.5925415Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:15:06.5929744Z test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-08-17T12:15:06.5932756Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:15:06.5986670Z test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6011 2022-08-17T12:15:06.6027549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6012 2022-08-17T12:15:06.6060186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6013 2022-08-17T12:15:06.6101245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6014 2022-08-17T12:15:07.6571975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:07.6572643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:07.6573647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:07.6574284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:07.6734062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:07.6734720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:07.6735771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:07.6736387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:07.6739861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:07.6740691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:07.6743503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:07.6744113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:07.7078026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:07.7078711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:07.7080099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:07.7080746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:07.8185183Z dist init r=1, world=4 2022-08-17T12:15:07.8337506Z dist init r=3, world=4 2022-08-17T12:15:07.8535568Z dist init r=2, world=4 2022-08-17T12:15:07.8666338Z dist init r=0, world=4 2022-08-17T12:15:07.9079082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:07.9180389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:07.9282899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:07.9283931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:07.9284364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:07.9284902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:07.9285447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:07.9286216Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:07.9292363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:07.9292838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:07.9293392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:07.9293974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:08.2139175Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:08.2193158Z test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6067 2022-08-17T12:15:08.2229242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6068 2022-08-17T12:15:08.2257747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6069 2022-08-17T12:15:08.2286278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6070 2022-08-17T12:15:09.2310421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:09.2311118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:09.2312248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:09.2312842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:09.2634834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:09.2635359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:09.2636026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:09.2636776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:09.2637453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:09.2637974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:09.2641366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:09.2642079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:09.2664061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:09.2664696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:09.2667594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:09.2668256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:09.3885079Z dist init r=3, world=4 2022-08-17T12:15:09.4227629Z dist init r=1, world=4 2022-08-17T12:15:09.4238417Z dist init r=2, world=4 2022-08-17T12:15:09.4245526Z dist init r=0, world=4 2022-08-17T12:15:09.4447316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:09.4548403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:09.4650349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:09.4651291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:09.4652198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:09.4653099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:09.4653926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:09.4654795Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:09.4658018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:09.4658536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:09.4659071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:09.4659619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:09.8327480Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:09.8382260Z test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6123 2022-08-17T12:15:09.8418845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6124 2022-08-17T12:15:09.8447222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6125 2022-08-17T12:15:09.8476012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6126 2022-08-17T12:15:10.8977816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:10.8978520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:10.8979523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:10.8980164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:10.9225514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:10.9226415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:10.9229288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:10.9229849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:10.9851007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:10.9851660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:10.9853750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:10.9854348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:10.9942443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:10.9943054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:10.9945487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:10.9946321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:11.0597494Z dist init r=0, world=4 2022-08-17T12:15:11.0994561Z dist init r=1, world=4 2022-08-17T12:15:11.1436953Z dist init r=3, world=4 2022-08-17T12:15:11.1502907Z dist init r=2, world=4 2022-08-17T12:15:11.1646465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:11.1747262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:11.1848282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:11.1848864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:11.1849889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:11.1850711Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:11.1851384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:11.1851989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:11.1955982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:11.1956688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:11.1957105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:11.1957468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:11.5517486Z skip: Need at least 2 CUDA devices (1.719s) 2022-08-17T12:15:11.5572055Z test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6179 2022-08-17T12:15:11.5611475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6180 2022-08-17T12:15:11.5640669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6181 2022-08-17T12:15:11.5671005Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6182 2022-08-17T12:15:12.5556539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:12.5557203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:12.5558279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:12.5559144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:12.5710947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:12.5711668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:12.5712694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:12.5713310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:12.6062737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:12.6063423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:12.6064414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:12.6065005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:12.6186560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:12.6187485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:12.6188790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:12.6189467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:12.7172082Z dist init r=0, world=4 2022-08-17T12:15:12.7345921Z dist init r=2, world=4 2022-08-17T12:15:12.7678097Z dist init r=3, world=4 2022-08-17T12:15:12.7782780Z dist init r=1, world=4 2022-08-17T12:15:12.8060644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:12.8161035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:12.8263029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:12.8264536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:12.8265989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:12.8266755Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:12.8267378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:12.8268131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:12.8271141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:12.8271747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:12.8272909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:12.8273971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:13.1711392Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:13.1766244Z test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6235 2022-08-17T12:15:13.1805422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6236 2022-08-17T12:15:13.1837727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6237 2022-08-17T12:15:13.1876508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6238 2022-08-17T12:15:14.1929688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:14.1930327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:14.1931593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:14.1932171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:14.1995372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:14.1996011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:14.1997536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:14.1998180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:14.2408296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:14.2408924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:14.2410172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:14.2411030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:14.2602533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:14.2603677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:14.2604804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:14.2605423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:14.3536544Z dist init r=1, world=4 2022-08-17T12:15:14.3590693Z dist init r=3, world=4 2022-08-17T12:15:14.4016035Z dist init r=0, world=4 2022-08-17T12:15:14.4203425Z dist init r=2, world=4 2022-08-17T12:15:14.4403306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:14.4503791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:14.4606328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:14.4606955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:14.4607978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:14.4608604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:14.4609153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:14.4609695Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:14.4714731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:14.4715444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:14.4715813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:14.4716164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:14.7916290Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:14.7975995Z test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6291 2022-08-17T12:15:14.8014526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6292 2022-08-17T12:15:14.8045326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6293 2022-08-17T12:15:14.8076019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6294 2022-08-17T12:15:15.8446698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:15.8447367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:15.8448183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:15.8448869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:15.8915017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:15.8915888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:15.8916747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:15.8917427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:15.9178740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:15.9179894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:15.9181030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:15.9181662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:15.9331676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:15.9332303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:15.9333544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:15.9334138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:16.0030971Z dist init r=3, world=4 2022-08-17T12:15:16.0515364Z dist init r=0, world=4 2022-08-17T12:15:16.0769924Z dist init r=1, world=4 2022-08-17T12:15:16.0924338Z dist init r=2, world=4 2022-08-17T12:15:16.1181114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:16.1283271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:16.1283919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:16.1284929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:16.1285680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:16.1286472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:16.1287705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:16.1288851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:16.1290387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:16.1291184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:16.1291920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:16.1292647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:16.5116937Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:15:16.5171706Z test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6347 2022-08-17T12:15:16.5209565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6348 2022-08-17T12:15:16.5240068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6349 2022-08-17T12:15:16.5269864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6350 2022-08-17T12:15:17.5836307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:17.5836991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:17.5837828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:17.5838469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:17.5956951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:17.5957566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:17.5959297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:17.5959891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:17.5996114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:17.5996729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:17.5998358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:17.5998989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:17.6359232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:17.6359918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:17.6360880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:17.6361602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:17.7445270Z dist init r=0, world=4 2022-08-17T12:15:17.7557227Z dist init r=3, world=4 2022-08-17T12:15:17.7606318Z dist init r=1, world=4 2022-08-17T12:15:17.7935143Z dist init r=2, world=4 2022-08-17T12:15:17.8069225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:17.8169755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:17.8219567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:17.8219988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:17.8220701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:17.8221367Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:17.8272659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:17.8273411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:17.8378634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:17.8379326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:17.8379997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:17.8380837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:18.1310028Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:18.1364419Z test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6403 2022-08-17T12:15:18.1400651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6404 2022-08-17T12:15:18.1429824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6405 2022-08-17T12:15:18.1459776Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6406 2022-08-17T12:15:19.1543188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:19.1543853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:19.1547165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:19.1547816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:19.1920222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:19.1920825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:19.1922333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:19.1922953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:19.1991346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:19.1991907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:19.1993195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:19.1993874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:19.2309313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:19.2310071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:19.2312349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:19.2332893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:19.3149415Z dist init r=2, world=4 2022-08-17T12:15:19.3554951Z dist init r=0, world=4 2022-08-17T12:15:19.3601290Z dist init r=3, world=4 2022-08-17T12:15:19.3890369Z dist init r=1, world=4 2022-08-17T12:15:19.4012921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:19.4203984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:19.4204602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:19.4205155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:19.4206264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:19.4207025Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:19.4207805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:19.4216609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:19.4312137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:19.4312745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:19.4313249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:19.4313821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:19.7498832Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:19.7554156Z test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6459 2022-08-17T12:15:19.7591407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6460 2022-08-17T12:15:19.7620001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6461 2022-08-17T12:15:19.7650218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6462 2022-08-17T12:15:20.7665677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:20.7666500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:20.7667631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:20.7668300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:20.7957817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:20.7958454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:20.7959260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:20.7959854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:20.8065983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:20.8066681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:20.8067487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:20.8068049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:20.8075520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:20.8076174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:20.8078767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:20.8079391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:20.9269970Z dist init r=0, world=4 2022-08-17T12:15:20.9572091Z dist init r=1, world=4 2022-08-17T12:15:20.9687321Z dist init r=3, world=4 2022-08-17T12:15:20.9687684Z dist init r=2, world=4 2022-08-17T12:15:20.9798203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:20.9897255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:20.9999596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:21.0000109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:21.0000867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:21.0001625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:21.0002460Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:21.0003013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:21.0108149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:21.0108796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:21.0109270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:21.0109650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:21.3692222Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:21.3747401Z test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6515 2022-08-17T12:15:21.3787137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6516 2022-08-17T12:15:21.3818013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6517 2022-08-17T12:15:21.3848743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6518 2022-08-17T12:15:22.4129894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:22.4130618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:22.4131643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:22.4132320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:22.4248561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:22.4249233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:22.4250563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:22.4251210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:22.4275683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:22.4276478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:22.4278948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:22.4279636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:22.4335855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:22.4336596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:22.4338017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:22.4338719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:22.5857452Z dist init r=2, world=4 2022-08-17T12:15:22.5890430Z dist init r=3, world=4 2022-08-17T12:15:22.5941961Z dist init r=1, world=4 2022-08-17T12:15:22.5943813Z dist init r=0, world=4 2022-08-17T12:15:22.6358074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:22.6458953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:22.6561733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:22.6562844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:22.6563593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:22.6564343Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:22.6564890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:22.6565441Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:22.6568892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:22.6569484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:22.6569991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:22.6571857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:22.9887835Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:22.9942651Z test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6571 2022-08-17T12:15:22.9981072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6572 2022-08-17T12:15:23.0009413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6573 2022-08-17T12:15:23.0039148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6574 2022-08-17T12:15:23.9838642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:23.9839328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:23.9840853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:23.9841579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:24.0343799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:24.0344489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:24.0346255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:24.0346906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:24.0497066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:24.0497778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:24.0499199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:24.0499859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:24.0691065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:24.0691746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:24.0692828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:24.0693445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:24.1419867Z dist init r=3, world=4 2022-08-17T12:15:24.1932098Z dist init r=1, world=4 2022-08-17T12:15:24.2093504Z dist init r=0, world=4 2022-08-17T12:15:24.2260154Z dist init r=2, world=4 2022-08-17T12:15:24.2432547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:24.2571356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:24.2673085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:24.2673814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:24.2674538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:24.2675086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:24.2675642Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:24.2736299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:24.2780665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:24.2781265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:24.2781874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:24.2782423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:24.6078678Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:24.6133309Z test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6627 2022-08-17T12:15:24.6170855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6628 2022-08-17T12:15:24.6199081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6629 2022-08-17T12:15:24.6228404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6630 2022-08-17T12:15:25.6349435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:25.6350108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:25.6351868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:25.6352473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:25.6430112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:25.6430776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:25.6432586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:25.6433159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:25.6633009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:25.6633692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:25.6634953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:25.6635550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:25.6792677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:25.6793370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:25.6794636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:25.6795276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:25.8065963Z dist init r=3, world=4 2022-08-17T12:15:25.8111056Z dist init r=0, world=4 2022-08-17T12:15:25.8253492Z dist init r=2, world=4 2022-08-17T12:15:25.8385885Z dist init r=1, world=4 2022-08-17T12:15:25.8724609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:25.8825929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:25.8829222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:25.8829884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:25.8830753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:25.8831675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:25.8832558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:25.8833316Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:25.8834057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:25.8834657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:25.8835429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:25.8836788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:26.2269356Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:26.2324201Z test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6683 2022-08-17T12:15:26.2362682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6684 2022-08-17T12:15:26.2392467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6685 2022-08-17T12:15:26.2422887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6686 2022-08-17T12:15:27.2184454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:27.2185825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:27.2186991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:27.2187624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:27.2188434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:27.2189012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:27.2189832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:27.2190455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:27.2393333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:27.2394072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:27.2395672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:27.2396368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:27.3008382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:27.3009024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:27.3010268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:27.3010896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:27.3772762Z dist init r=1, world=4 2022-08-17T12:15:27.3780550Z dist init r=0, world=4 2022-08-17T12:15:27.3992838Z dist init r=2, world=4 2022-08-17T12:15:27.4571530Z dist init r=3, world=4 2022-08-17T12:15:27.4705202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:27.4807375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:27.4907484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:27.4908402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:27.4909386Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:27.4911557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:27.4912280Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:27.4912996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:27.5015470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:27.5015898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:27.5016454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:27.5017015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:27.8462746Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:27.8513104Z test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6739 2022-08-17T12:15:27.8545762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6740 2022-08-17T12:15:27.8574108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6741 2022-08-17T12:15:27.8602664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6742 2022-08-17T12:15:28.9078131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:28.9078863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:28.9079713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:28.9080375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:28.9147426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:28.9148125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:28.9149759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:28.9150446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:28.9190069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:28.9190654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:28.9193109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:28.9193689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:28.9208851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:28.9209715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:28.9212224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:28.9212859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:29.0686639Z dist init r=2, world=4 2022-08-17T12:15:29.0759642Z dist init r=0, world=4 2022-08-17T12:15:29.0805342Z dist init r=3, world=4 2022-08-17T12:15:29.0818540Z dist init r=1, world=4 2022-08-17T12:15:29.1015505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:29.1115858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:29.1172195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:29.1172769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:29.1173436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:29.1173987Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:29.1218786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:29.1219376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:29.1280473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:29.1281177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:29.1281743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:29.1282309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:29.4641836Z skip: Need at least 2 CUDA devices (1.618s) 2022-08-17T12:15:29.4696871Z test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6795 2022-08-17T12:15:29.4735431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6796 2022-08-17T12:15:29.4766090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6797 2022-08-17T12:15:29.4796743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6798 2022-08-17T12:15:30.4852826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:30.4853459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:30.4854831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:30.4855492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:30.4891039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:30.4891701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:30.4894457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:30.4895062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:30.5200864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:30.5202021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:30.5203023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:30.5203843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:30.5204529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:30.5205032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:30.5208399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:30.5208996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:30.6445233Z dist init r=1, world=4 2022-08-17T12:15:30.6467313Z dist init r=2, world=4 2022-08-17T12:15:30.6775523Z dist init r=0, world=4 2022-08-17T12:15:30.6790432Z dist init r=3, world=4 2022-08-17T12:15:30.6977608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:30.7079708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:30.7101749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:30.7102374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:30.7103215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:30.7103771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:30.7180985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:30.7181875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:30.7210390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:30.7211005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:30.7211593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:30.7212139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:31.0835557Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:31.0891054Z test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6851 2022-08-17T12:15:31.0928306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6852 2022-08-17T12:15:31.0956965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6853 2022-08-17T12:15:31.0986340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6854 2022-08-17T12:15:32.0814978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:32.0815639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:32.0816613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:32.0817180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:32.0918538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:32.0919762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:32.0920843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:32.0921463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:32.1003787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:32.1004711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:32.1007177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:32.1007889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:32.1647427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:32.1648054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:32.1648895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:32.1649536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:32.2427986Z dist init r=0, world=4 2022-08-17T12:15:32.2512154Z dist init r=3, world=4 2022-08-17T12:15:32.2589502Z dist init r=1, world=4 2022-08-17T12:15:32.3200016Z dist init r=2, world=4 2022-08-17T12:15:32.3447596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:32.3448192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:32.3448869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:32.3449528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:32.3450056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:32.3450839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:32.3451702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:32.3452497Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:32.3453753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:32.3454405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:32.3455026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:32.3455584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:32.7026874Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:32.7082503Z test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6907 2022-08-17T12:15:32.7121427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6908 2022-08-17T12:15:32.7150693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6909 2022-08-17T12:15:32.7179983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6910 2022-08-17T12:15:33.7028706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:33.7029366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:33.7030482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:33.7031069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:33.7215608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:33.7216265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:33.7217275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:33.7218162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:33.7237962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:33.7238638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:33.7240286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:33.7240969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:33.7716992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:33.7717692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:33.7718717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:33.7719393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:33.8623547Z dist init r=0, world=4 2022-08-17T12:15:33.8810538Z dist init r=3, world=4 2022-08-17T12:15:33.8842329Z dist init r=1, world=4 2022-08-17T12:15:33.9285022Z dist init r=2, world=4 2022-08-17T12:15:33.9495491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:33.9537529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:33.9538061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:33.9538886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:33.9539307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:33.9539846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:33.9540390Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:33.9597021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:33.9645556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:33.9646391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:33.9646976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:33.9647563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:34.3218838Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:34.3275008Z test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6963 2022-08-17T12:15:34.3313209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6964 2022-08-17T12:15:34.3343915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6965 2022-08-17T12:15:34.3373613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6966 2022-08-17T12:15:35.3758724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:35.3759375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:35.3760715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:35.3761279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:35.3815648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:35.3816352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:35.3817203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:35.3817806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:35.3818937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:35.3819612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:35.3820437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:35.3821066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:35.3885115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:35.3885788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:35.3887611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:35.3888411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:35.5360246Z dist init r=3, world=4 2022-08-17T12:15:35.5422765Z dist init r=1, world=4 2022-08-17T12:15:35.5423698Z dist init r=2, world=4 2022-08-17T12:15:35.5488270Z dist init r=0, world=4 2022-08-17T12:15:35.5670089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:35.5769934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:35.5873066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:35.5873647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:35.5874306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:35.5874870Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:35.5875419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:35.5875964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:35.5979451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:35.5980171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:35.5980856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:35.5981962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:35.9413754Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:35.9471818Z test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7019 2022-08-17T12:15:35.9509396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7020 2022-08-17T12:15:35.9539974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7021 2022-08-17T12:15:35.9570105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7022 2022-08-17T12:15:37.0269073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:37.0269731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:37.0270768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:37.0271443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:37.0452502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:37.0453199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:37.0454006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:37.0454648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:37.0455468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:37.0456047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:37.0461456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:37.0462131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:37.0598051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:37.0599006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:37.0600477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:37.0601122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:37.1863783Z dist init r=1, world=4 2022-08-17T12:15:37.2064302Z dist init r=2, world=4 2022-08-17T12:15:37.2064642Z dist init r=3, world=4 2022-08-17T12:15:37.2196633Z dist init r=0, world=4 2022-08-17T12:15:37.2575070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:37.2676672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:37.2778447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:37.2779146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:37.2780102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:37.2780894Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:37.2781581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:37.2782128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:37.2886135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:37.2886712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:37.2887086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:37.2887441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:37.6612622Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:15:37.6667197Z test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7075 2022-08-17T12:15:37.6705406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7076 2022-08-17T12:15:37.6735281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7077 2022-08-17T12:15:37.6764992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7078 2022-08-17T12:15:38.7085784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:38.7086658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:38.7087489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:38.7088029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:38.7090017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:38.7090611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:38.7091415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:38.7092040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:38.7112977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:38.7113627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:38.7114734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:38.7115554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:38.7144593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:38.7145216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:38.7147780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:38.7148355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:38.8710816Z dist init r=3, world=4 2022-08-17T12:15:38.8711185Z dist init r=2, world=4 2022-08-17T12:15:38.8755063Z dist init r=0, world=4 2022-08-17T12:15:38.8763423Z dist init r=1, world=4 2022-08-17T12:15:38.9076402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:38.9177372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:38.9279716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:38.9280375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:38.9281363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:38.9282184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:38.9283130Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:38.9283980Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:38.9291468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:38.9292080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:38.9292632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:38.9293187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:39.2804293Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:39.2860451Z test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7131 2022-08-17T12:15:39.2897826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7132 2022-08-17T12:15:39.2928406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7133 2022-08-17T12:15:39.2959479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7134 2022-08-17T12:15:40.3552314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:40.3552936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:40.3554013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:40.3554629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:40.3774709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:40.3775372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:40.3776467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:40.3777137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:40.3791432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:40.3792336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:40.3794098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:40.3794767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:40.3839295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:40.3839865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:40.3842081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:40.3842689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:40.5299590Z dist init r=1, world=4 2022-08-17T12:15:40.5395668Z dist init r=0, world=4 2022-08-17T12:15:40.5414822Z dist init r=2, world=4 2022-08-17T12:15:40.5473344Z dist init r=3, world=4 2022-08-17T12:15:40.5580499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:40.5681934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:40.5725311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:40.5726056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:40.5726705Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:40.5727276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:40.5784311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:40.5784915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:40.5833034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:40.5833758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:40.5834142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:40.5834507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:40.8999221Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:40.9053247Z test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7187 2022-08-17T12:15:40.9090169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7188 2022-08-17T12:15:40.9119037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7189 2022-08-17T12:15:40.9149515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7190 2022-08-17T12:15:41.8971818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:41.8972489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:41.8973345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:41.8973986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:41.9024087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:41.9024754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:41.9026643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:41.9027289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:41.9517978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:41.9518693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:41.9520194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:41.9520820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:41.9602679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:41.9603397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:41.9604737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:41.9605368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:42.0559392Z dist init r=3, world=4 2022-08-17T12:15:42.0630704Z dist init r=2, world=4 2022-08-17T12:15:42.1127488Z dist init r=1, world=4 2022-08-17T12:15:42.1199850Z dist init r=0, world=4 2022-08-17T12:15:42.1540994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:42.1642021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:42.1642721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:42.1643606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:42.1644350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:42.1644914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:42.1645464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:42.1646177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:42.1749982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:42.1750609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:42.1751350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:42.1752202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:42.5191095Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:42.5244185Z test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7243 2022-08-17T12:15:42.5282610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7244 2022-08-17T12:15:42.5313087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7245 2022-08-17T12:15:42.5353245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7246 2022-08-17T12:15:43.5425809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:43.5426466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:43.5427311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:43.5427963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:43.5577222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:43.5577912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:43.5579347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:43.5579937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:43.5764775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:43.5765438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:43.5767102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:43.5767666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:43.5821235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:43.5821653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:43.5823883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:43.5824392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:43.7035620Z dist init r=2, world=4 2022-08-17T12:15:43.7175090Z dist init r=3, world=4 2022-08-17T12:15:43.7408105Z dist init r=0, world=4 2022-08-17T12:15:43.7479764Z dist init r=1, world=4 2022-08-17T12:15:43.7686657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:43.7849682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:43.7951254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:43.7951840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:43.7952501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:43.7953048Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:43.7953606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:43.7989923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:43.8059022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:43.8059935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:43.8060514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:43.8061322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:44.1392080Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:44.1445744Z test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7299 2022-08-17T12:15:44.1483231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7300 2022-08-17T12:15:44.1512636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7301 2022-08-17T12:15:44.1541996Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7302 2022-08-17T12:15:45.1574582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:45.1575281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:45.1576280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:45.1576934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:45.1703754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:45.1704430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:45.1706589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:45.1707284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:45.1768180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:45.1768857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:45.1770853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:45.1771478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:45.1958884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:45.1959462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:45.1960775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:45.1961369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:45.3197490Z dist init r=2, world=4 2022-08-17T12:15:45.3319806Z dist init r=3, world=4 2022-08-17T12:15:45.3370356Z dist init r=0, world=4 2022-08-17T12:15:45.3557859Z dist init r=1, world=4 2022-08-17T12:15:45.3809049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:45.3909733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:45.4011982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:45.4012684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:45.4013797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:45.4014797Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:45.4015474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:45.4016192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:45.4024458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:45.4025234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:45.4025834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:45.4026393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:45.7581034Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:45.7636354Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7355 2022-08-17T12:15:45.7674346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7356 2022-08-17T12:15:45.7703489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7357 2022-08-17T12:15:45.7733119Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7358 2022-08-17T12:15:46.7690103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:46.7690785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:46.7691590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:46.7692173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:46.7730897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:46.7731532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:46.7734337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:46.7735120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:46.8149626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:46.8150279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:46.8151134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:46.8151869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:46.8762913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:46.8763541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:46.8764343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:46.8764943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:46.9299290Z dist init r=2, world=4 2022-08-17T12:15:46.9357972Z dist init r=3, world=4 2022-08-17T12:15:46.9746006Z dist init r=0, world=4 2022-08-17T12:15:47.0339089Z dist init r=1, world=4 2022-08-17T12:15:47.0615454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:47.0716660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:47.0819441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:47.0820088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:47.0821061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:47.0822105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:47.0822992Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:47.0823849Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:47.0826275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:47.0826833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:47.0827920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:47.0828334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:47.3775109Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:47.3831708Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7411 2022-08-17T12:15:47.3871672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7412 2022-08-17T12:15:47.3902303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7413 2022-08-17T12:15:47.3933055Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7414 2022-08-17T12:15:48.4117586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:48.4118230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:48.4119036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:48.4119623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:48.4121871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:48.4122499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:48.4123292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:48.4123957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:48.4329981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:48.4330693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:48.4332499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:48.4333195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:48.4468952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:48.4469523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:48.4472032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:48.4472597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:48.5742168Z dist init r=3, world=4 2022-08-17T12:15:48.5746744Z dist init r=1, world=4 2022-08-17T12:15:48.5932398Z dist init r=0, world=4 2022-08-17T12:15:48.6097566Z dist init r=2, world=4 2022-08-17T12:15:48.6361570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:48.6462583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:48.6463322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:48.6464482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:48.6465141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:48.6465897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:48.6466452Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:48.6467007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:48.6571002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:48.6571497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:48.6572241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:48.6572932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:48.9972499Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:49.0029936Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7467 2022-08-17T12:15:49.0069611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7468 2022-08-17T12:15:49.0100079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7469 2022-08-17T12:15:49.0130593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7470 2022-08-17T12:15:49.9974917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:49.9975553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:49.9976316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:49.9976947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:50.0505883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:50.0506775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:50.0507697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:50.0508422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:50.0527512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:50.0528220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:50.0530444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:50.0531178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:50.0540312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:50.0541039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:50.0543382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:50.0544105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:50.1563736Z dist init r=2, world=4 2022-08-17T12:15:50.2115108Z dist init r=3, world=4 2022-08-17T12:15:50.2148792Z dist init r=0, world=4 2022-08-17T12:15:50.2150464Z dist init r=1, world=4 2022-08-17T12:15:50.2561904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:50.2663046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:50.2766412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:50.2767418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:50.2768274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:50.2768782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:50.2769312Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:50.2769846Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:50.2874511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:50.2875118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:50.2877266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:50.2877795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:50.6171511Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:50.6227307Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7523 2022-08-17T12:15:50.6266797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7524 2022-08-17T12:15:50.6297259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7525 2022-08-17T12:15:50.6327668Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7526 2022-08-17T12:15:51.6558786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:51.6559540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:51.6560395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:51.6561029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:51.6738566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:51.6739289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:51.6740056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:51.6740670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:51.6745196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:51.6745810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:51.6748794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:51.6749480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:51.7051919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:51.7052704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:51.7053533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:51.7054178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:51.8179660Z dist init r=2, world=4 2022-08-17T12:15:51.8345199Z dist init r=1, world=4 2022-08-17T12:15:51.8364619Z dist init r=0, world=4 2022-08-17T12:15:51.8649897Z dist init r=3, world=4 2022-08-17T12:15:51.8858476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:51.8859030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:51.8859617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:51.8860236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:51.8861081Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:51.8861685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:51.8862246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:51.8862791Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:51.8966678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:51.8967369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:51.8967924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:51.8968491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:52.2368438Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:52.2425444Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7579 2022-08-17T12:15:52.2463095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7580 2022-08-17T12:15:52.2493417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7581 2022-08-17T12:15:52.2524913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7582 2022-08-17T12:15:53.2373809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:53.2374448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:53.2375567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:53.2376156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:53.2572948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:53.2573674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:53.2575425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:53.2576123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:53.2851426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:53.2852072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:53.2853421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:53.2854073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:53.2868458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:53.2869257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:53.2872160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:53.2872715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:53.3952648Z dist init r=3, world=4 2022-08-17T12:15:53.4168520Z dist init r=1, world=4 2022-08-17T12:15:53.4441648Z dist init r=0, world=4 2022-08-17T12:15:53.4456339Z dist init r=2, world=4 2022-08-17T12:15:53.4664620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:53.4764386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:53.4867854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:53.4868978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:53.4869737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:53.4870609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:53.4871386Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:53.4871943Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:53.4974364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:53.4975102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:53.4975661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:53.4976254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:53.8564920Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:53.8621648Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7635 2022-08-17T12:15:53.8659353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7636 2022-08-17T12:15:53.8688898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7637 2022-08-17T12:15:53.8718760Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7638 2022-08-17T12:15:54.9224685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:54.9225371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:54.9226963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:54.9227693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:54.9289281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:54.9289918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:54.9292664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:54.9293356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:54.9377934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:54.9378583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:54.9380921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:54.9381825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:54.9621908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:54.9622487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:54.9623905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:54.9624440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:55.0826223Z dist init r=2, world=4 2022-08-17T12:15:55.0885476Z dist init r=1, world=4 2022-08-17T12:15:55.0974447Z dist init r=3, world=4 2022-08-17T12:15:55.1203074Z dist init r=0, world=4 2022-08-17T12:15:55.1440042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:55.1541022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:55.1643999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:55.1644584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:55.1645524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:55.1646374Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:55.1646925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:55.1647708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:55.1750384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:55.1750961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:55.1751629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:55.1752248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:55.4759114Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:55.4815496Z test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7691 2022-08-17T12:15:55.4852292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7692 2022-08-17T12:15:55.4881311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7693 2022-08-17T12:15:55.4911035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7694 2022-08-17T12:15:56.4704636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:56.4705327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:56.4706190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:56.4706875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:56.5160002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:56.5160581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:56.5162347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:56.5162937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:56.5222445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:56.5223385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:56.5224940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:56.5225588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:56.5486854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:56.5487431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:56.5489458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:56.5489981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:56.6298942Z dist init r=2, world=4 2022-08-17T12:15:56.6751846Z dist init r=1, world=4 2022-08-17T12:15:56.6821229Z dist init r=0, world=4 2022-08-17T12:15:56.7074759Z dist init r=3, world=4 2022-08-17T12:15:56.7182367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:56.7213728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:56.7316058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:56.7316580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:56.7317245Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:56.7317808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:56.7318362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:56.7385315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:56.7422950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:56.7423558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:56.7424166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:56.7424669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:57.0951889Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:15:57.1008035Z test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7747 2022-08-17T12:15:57.1046533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7748 2022-08-17T12:15:57.1077780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7749 2022-08-17T12:15:57.1108937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7750 2022-08-17T12:15:58.1291920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:58.1292594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:58.1293799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:58.1294410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:58.1546577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:58.1547181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:58.1547999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:58.1548887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:58.1665834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:58.1666475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:58.1667878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:58.1668574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:58.2209210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:58.2209893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:58.2211439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:58.2212055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:58.2907514Z dist init r=3, world=4 2022-08-17T12:15:58.3256053Z dist init r=2, world=4 2022-08-17T12:15:58.3362934Z dist init r=1, world=4 2022-08-17T12:15:58.3794941Z dist init r=0, world=4 2022-08-17T12:15:58.4208013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:58.4208804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:58.4210219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:58.4210688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:58.4211079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:58.4211613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:58.4212406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:58.4213193Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:58.4219324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:58.4219823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:58.4220386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:15:58.4220948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:58.7148624Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:15:58.7205742Z test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7803 2022-08-17T12:15:58.7243158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7804 2022-08-17T12:15:58.7273605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7805 2022-08-17T12:15:58.7303927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7806 2022-08-17T12:15:59.7279911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:59.7280529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:59.7285362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:59.7286391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:59.7288510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:59.7289602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:59.7291253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:59.7292047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:59.7629700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:59.7630266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:59.7631549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:59.7632039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:59.7804003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:15:59.7804888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:15:59.7806244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:15:59.7807172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:15:59.8880315Z dist init r=2, world=4 2022-08-17T12:15:59.8900582Z dist init r=1, world=4 2022-08-17T12:15:59.9226874Z dist init r=3, world=4 2022-08-17T12:15:59.9383406Z dist init r=0, world=4 2022-08-17T12:15:59.9693366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:15:59.9794702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:15:59.9898092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:15:59.9898982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:59.9899605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:15:59.9900365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:59.9901215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:59.9901913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:15:59.9904956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:15:59.9905582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:15:59.9906408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:15:59.9909757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:00.3344749Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:00.3400130Z test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7859 2022-08-17T12:16:00.3437232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7860 2022-08-17T12:16:00.3466441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7861 2022-08-17T12:16:00.3496638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7862 2022-08-17T12:16:01.3403793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:01.3404473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:01.3405268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:01.3406394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:01.3528888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:01.3529557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:01.3530543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:01.3531162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:01.3848132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:01.3848674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:01.3850542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:01.3851066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:01.3858006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:01.3858817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:01.3861785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:01.3862320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:01.5078126Z dist init r=3, world=4 2022-08-17T12:16:01.5204561Z dist init r=0, world=4 2022-08-17T12:16:01.5488169Z dist init r=2, world=4 2022-08-17T12:16:01.5611957Z dist init r=1, world=4 2022-08-17T12:16:01.5799072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:01.6000420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:01.6102441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:01.6103156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:01.6103992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:01.6104555Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:01.6105105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:01.6105640Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:01.6209892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:01.6210311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:01.6210827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:01.6211386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:01.9535839Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:01.9591364Z test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7915 2022-08-17T12:16:01.9629454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7916 2022-08-17T12:16:01.9659883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7917 2022-08-17T12:16:01.9689643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7918 2022-08-17T12:16:02.9571352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:02.9572023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:02.9573190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:02.9573831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:02.9733482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:02.9734173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:02.9735417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:02.9736040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:02.9829623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:02.9830312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:02.9832546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:02.9833456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:03.0478101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:03.0478758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:03.0480248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:03.0480867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:03.1177074Z dist init r=0, world=4 2022-08-17T12:16:03.1370412Z dist init r=2, world=4 2022-08-17T12:16:03.1421702Z dist init r=1, world=4 2022-08-17T12:16:03.2047525Z dist init r=3, world=4 2022-08-17T12:16:03.2156259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:03.2184454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:03.2184970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:03.2185371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:03.2186012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:03.2186558Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:03.2187103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:03.2259005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:03.2292019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:03.2292593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:03.2293074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:03.2293603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:03.5730415Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:03.5785943Z test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7971 2022-08-17T12:16:03.5823693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7972 2022-08-17T12:16:03.5853246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7973 2022-08-17T12:16:03.5882539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7974 2022-08-17T12:16:04.5884850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:04.5885525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:04.5888253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:04.5888644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:04.6150303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:04.6150857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:04.6152073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:04.6152689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:04.6223471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:04.6224411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:04.6226678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:04.6227229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:04.6558322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:04.6558771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:04.6561595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:04.6562225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:04.7721228Z dist init r=2, world=4 2022-08-17T12:16:04.7733371Z dist init r=3, world=4 2022-08-17T12:16:04.7804739Z dist init r=0, world=4 2022-08-17T12:16:04.8142014Z dist init r=1, world=4 2022-08-17T12:16:04.8344893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:04.8517835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:04.8619535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:04.8620094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:04.8620736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:04.8621309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:04.8621866Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:04.8648692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:04.8726968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:04.8727677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:04.8728356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:04.8729009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:05.1923012Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:05.1923576Z 2022-08-17T12:16:05.1923982Z ---------------------------------------------------------------------- 2022-08-17T12:16:05.1924258Z Ran 59 tests in 88.124s 2022-08-17T12:16:05.1924447Z 2022-08-17T12:16:05.1924584Z OK (skipped=59) 2022-08-17T12:16:05.1924808Z 2022-08-17T12:16:05.1925140Z Generating XML reports... 2022-08-17T12:16:05.1980872Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220817121437.xml 2022-08-17T12:16:05.1985302Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220817121437.xml 2022-08-17T12:16:05.1988901Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220817121437.xml 2022-08-17T12:16:05.2038809Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220817121437.xml 2022-08-17T12:16:05.5256361Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-08-17 12:16:05.525240] 2022-08-17T12:16:05.5257067Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:16:05.525323] 2022-08-17T12:16:06.6565504Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2022-08-17T12:16:06.6578166Z 2022-08-17T12:16:06.6578353Z Running tests... 2022-08-17T12:16:06.6578989Z ---------------------------------------------------------------------- 2022-08-17T12:16:06.6586714Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-08-17T12:16:06.9279436Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8038 2022-08-17T12:16:06.9308372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8039 2022-08-17T12:16:06.9337540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8040 2022-08-17T12:16:06.9369256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8041 2022-08-17T12:16:07.9455532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:07.9456226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:07.9457103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:07.9457755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:07.9567853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:07.9568468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:07.9571146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:07.9571843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:07.9810374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:07.9811008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:07.9814046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:07.9814671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:08.0024518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:08.0025212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:08.0026632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:08.0027404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:08.1036047Z dist init r=1, world=4 2022-08-17T12:16:08.1156815Z dist init r=0, world=4 2022-08-17T12:16:08.1383216Z dist init r=3, world=4 2022-08-17T12:16:08.1598804Z dist init r=2, world=4 2022-08-17T12:16:08.1794031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:08.1894956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:08.1973191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:08.1973643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:08.1974337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:08.1974959Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:08.1997771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:08.1998512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:08.2080497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:08.2081033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:08.2081612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:08.2082161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:08.5412444Z skip: Need at least 2 CUDA devices (1.883s) 2022-08-17T12:16:08.5420303Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-08-17T12:16:08.5467998Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8094 2022-08-17T12:16:08.5504573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8095 2022-08-17T12:16:08.5533003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8096 2022-08-17T12:16:08.5562761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8097 2022-08-17T12:16:09.5535117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:09.5535774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:09.5536836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:09.5537490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:09.6055077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:09.6056068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:09.6057167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:09.6057799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:09.6292552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:09.6293207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:09.6294413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:09.6295051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:09.6346809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:09.6347749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:09.6349734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:09.6350389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:09.7101408Z dist init r=1, world=4 2022-08-17T12:16:09.7624007Z dist init r=0, world=4 2022-08-17T12:16:09.7910245Z dist init r=2, world=4 2022-08-17T12:16:09.7951904Z dist init r=3, world=4 2022-08-17T12:16:09.8063709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:09.8164780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:09.8266102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:09.8266892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:09.8267959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:09.8268694Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:09.8269395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:09.8269952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:09.8373256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:09.8373932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:09.8374375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:09.8374744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:10.1603085Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:10.1617509Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-08-17T12:16:10.1665329Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8150 2022-08-17T12:16:10.1703461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8151 2022-08-17T12:16:10.1732904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8152 2022-08-17T12:16:10.1762147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8153 2022-08-17T12:16:11.1776983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:11.1777669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:11.1778496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:11.1779130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:11.1779940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:11.1780560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:11.1781409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:11.1782059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:11.2257347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:11.2258326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:11.2259361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:11.2259986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:11.2337447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:11.2338052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:11.2341302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:11.2342069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:11.3376133Z dist init r=1, world=4 2022-08-17T12:16:11.3391648Z dist init r=2, world=4 2022-08-17T12:16:11.3857231Z dist init r=0, world=4 2022-08-17T12:16:11.3917419Z dist init r=3, world=4 2022-08-17T12:16:11.4131351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:11.4131767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:11.4232914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:11.4233625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:11.4234414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:11.4234977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:11.4235510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:11.4236057Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:11.4341125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:11.4342129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:11.4342793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:11.4343434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:11.7803471Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:11.7818533Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-08-17T12:16:11.7868134Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8206 2022-08-17T12:16:11.7909177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8207 2022-08-17T12:16:11.7939740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8208 2022-08-17T12:16:11.7970547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8209 2022-08-17T12:16:12.8746036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:12.8746613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:12.8749503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:12.8750254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:12.8901612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:12.8902149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:12.8905696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:12.8906324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:12.8997714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:12.8998371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:12.9000005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:12.9000643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:12.9416834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:12.9417493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:12.9419039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:12.9419678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:13.0560012Z dist init r=2, world=4 2022-08-17T12:16:13.0566183Z dist init r=0, world=4 2022-08-17T12:16:13.0704993Z dist init r=1, world=4 2022-08-17T12:16:13.0992089Z dist init r=3, world=4 2022-08-17T12:16:13.1119154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:13.1120249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:13.1120789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:13.1121191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:13.1121959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:13.1122772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:13.1123622Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:13.1124261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:13.1126985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:13.1127533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:13.1127997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:13.1128372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:13.4011438Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:16:13.4026868Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-08-17T12:16:13.4076173Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8262 2022-08-17T12:16:13.4113423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8263 2022-08-17T12:16:13.4142291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8264 2022-08-17T12:16:13.4172246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8265 2022-08-17T12:16:14.4514882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:14.4515526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:14.4517093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:14.4517956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:14.4582922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:14.4583620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:14.4586256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:14.4586936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:14.5523084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:14.5523769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:14.5524844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:14.5525507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:14.5669798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:14.5670753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:14.5672582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:14.5673230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:14.6109292Z dist init r=1, world=4 2022-08-17T12:16:14.6166981Z dist init r=2, world=4 2022-08-17T12:16:14.7111600Z dist init r=3, world=4 2022-08-17T12:16:14.7228038Z dist init r=0, world=4 2022-08-17T12:16:14.7422386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:14.7523695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:14.7625602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:14.7626788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:14.7627437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:14.7627962Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:14.7628520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:14.7629068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:14.7635929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:14.7636697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:14.7638671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:14.7639377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:15.1214131Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:16:15.1228086Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-08-17T12:16:15.1286596Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8318 2022-08-17T12:16:15.1323867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8319 2022-08-17T12:16:15.1352988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8320 2022-08-17T12:16:15.1382822Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8321 2022-08-17T12:16:16.1188902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:16.1189577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:16.1190675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:16.1191382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:16.1230308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:16.1231065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:16.1233929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:16.1234597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:16.1378971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:16.1381415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:16.1382517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:16.1383155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:16.1820335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:16.1821086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:16.1822469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:16.1823200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:16.2775858Z dist init r=3, world=4 2022-08-17T12:16:16.2848691Z dist init r=2, world=4 2022-08-17T12:16:16.2975452Z dist init r=0, world=4 2022-08-17T12:16:16.3391517Z dist init r=1, world=4 2022-08-17T12:16:16.3589119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:16.3690144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:16.3765564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:16.3766091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:16.3766738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:16.3767294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:16.3792959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:16.3793653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:16.3873222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:16.3873766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:16.3874321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:16.3875601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:16.7421584Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:16:16.7481661Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8374 2022-08-17T12:16:16.7520901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8375 2022-08-17T12:16:16.7549783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8376 2022-08-17T12:16:16.7580183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8377 2022-08-17T12:16:17.7782364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:17.7782984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:17.7784419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:17.7785717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:17.8045625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:17.8046525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:17.8048001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:17.8048628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:17.8152488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:17.8153128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:17.8154334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:17.8154975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:17.8316350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:17.8316917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:17.8319125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:17.8319785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:17.9389329Z dist init r=0, world=4 2022-08-17T12:16:17.9639819Z dist init r=2, world=4 2022-08-17T12:16:17.9732605Z dist init r=3, world=4 2022-08-17T12:16:17.9875680Z dist init r=1, world=4 2022-08-17T12:16:18.0152243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:18.0204405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:18.0306567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:18.0307078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:18.0307823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:18.0308385Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:18.0308938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:18.0355301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:18.0413985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:18.0414554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:18.0415106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:18.0415755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:18.3619424Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:18.3681530Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8430 2022-08-17T12:16:18.3719325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8431 2022-08-17T12:16:18.3747330Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8432 2022-08-17T12:16:18.3775846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8433 2022-08-17T12:16:19.3548174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:19.3548890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:19.3549886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:19.3550543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:19.3596706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:19.3597338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:19.3599880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:19.3600504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:19.3994128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:19.3994814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:19.3995837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:19.3996481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:19.4257786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:19.4258335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:19.4260665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:19.4261233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:19.5139776Z dist init r=3, world=4 2022-08-17T12:16:19.5171341Z dist init r=2, world=4 2022-08-17T12:16:19.5571064Z dist init r=1, world=4 2022-08-17T12:16:19.5812370Z dist init r=0, world=4 2022-08-17T12:16:19.6086287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:19.6188153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:19.6289977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:19.6290974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:19.6291739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:19.6292475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:19.6293269Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:19.6294135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:19.6297852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:19.6298476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:19.6299109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:19.6299820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:19.9816530Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:19.9816860Z 2022-08-17T12:16:19.9817396Z ---------------------------------------------------------------------- 2022-08-17T12:16:19.9817804Z Ran 8 tests in 13.324s 2022-08-17T12:16:19.9817927Z 2022-08-17T12:16:19.9818006Z OK (skipped=8) 2022-08-17T12:16:19.9818103Z 2022-08-17T12:16:19.9818192Z Generating XML reports... 2022-08-17T12:16:19.9860934Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220817121606.xml 2022-08-17T12:16:20.3075606Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-08-17 12:16:20.307133] 2022-08-17T12:16:20.3076254Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:16:20.307219] 2022-08-17T12:16:21.4419089Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-08-17T12:16:21.4433343Z 2022-08-17T12:16:21.4433891Z Running tests... 2022-08-17T12:16:21.4434524Z ---------------------------------------------------------------------- 2022-08-17T12:16:21.7100423Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8497 2022-08-17T12:16:21.7129702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8498 2022-08-17T12:16:21.7159275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8499 2022-08-17T12:16:21.7190816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8500 2022-08-17T12:16:22.7901031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:22.7901660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:22.7902673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:22.7903264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:22.7904079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:22.7904649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:22.7907302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:22.7907933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:22.7915737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:22.7916377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:22.7919210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:22.7919847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:22.7935019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:22.7935605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:22.7938200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:22.7938828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:22.9506735Z dist init r=3, world=4 2022-08-17T12:16:22.9515117Z dist init r=0, world=4 2022-08-17T12:16:22.9520128Z dist init r=1, world=4 2022-08-17T12:16:22.9527222Z dist init r=2, world=4 2022-08-17T12:16:22.9739116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:22.9833518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:22.9834190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:22.9834946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:22.9835358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:22.9835884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:22.9836443Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:22.9840905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:22.9943037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:22.9943686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:22.9944287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:22.9944879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:23.3234410Z skip: Need at least 2 CUDA devices (1.880s) 2022-08-17T12:16:23.3288928Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8553 2022-08-17T12:16:23.3324550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8554 2022-08-17T12:16:23.3354789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8555 2022-08-17T12:16:23.3394152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8556 2022-08-17T12:16:24.3198907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:24.3199770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:24.3201208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:24.3201838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:24.3203724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:24.3204826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:24.3209120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:24.3209765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:24.3430803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:24.3433188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:24.3434102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:24.3434787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:24.4045575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:24.4046471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:24.4048420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:24.4049040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:24.4825076Z dist init r=2, world=4 2022-08-17T12:16:24.4838750Z dist init r=1, world=4 2022-08-17T12:16:24.5036457Z dist init r=0, world=4 2022-08-17T12:16:24.5608738Z dist init r=3, world=4 2022-08-17T12:16:24.5758380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:24.5759105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:24.5759808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:24.5760763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:24.5761218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:24.5761982Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:24.5762732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:24.5763288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:24.5767863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:24.5768570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:24.5770206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:24.5770801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:24.9434516Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:24.9491010Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8609 2022-08-17T12:16:24.9530254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8610 2022-08-17T12:16:24.9560005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8611 2022-08-17T12:16:24.9590216Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8612 2022-08-17T12:16:25.9349958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:25.9350600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:25.9352111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:25.9352779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:25.9518306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:25.9519017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:25.9520619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:25.9521933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:25.9893343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:25.9893942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:25.9895337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:25.9896247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:26.0022115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:26.0022831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:26.0024363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:26.0024935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:26.0940485Z dist init r=3, world=4 2022-08-17T12:16:26.1113555Z dist init r=1, world=4 2022-08-17T12:16:26.1484253Z dist init r=2, world=4 2022-08-17T12:16:26.1613894Z dist init r=0, world=4 2022-08-17T12:16:26.2027492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:26.2027908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:26.2130167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:26.2130903Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:26.2131535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:26.2132060Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:26.2132599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:26.2230476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:26.2337935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:26.2338530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:26.2339128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:26.2339605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:26.5630452Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:26.5686849Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8665 2022-08-17T12:16:26.5724555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8666 2022-08-17T12:16:26.5753586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8667 2022-08-17T12:16:26.5783933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8668 2022-08-17T12:16:27.5731621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:27.5732380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:27.5734045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:27.5734598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:27.5735341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:27.5735984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:27.5737701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:27.5738282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:27.5880089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:27.5880748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:27.5881559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:27.5882210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:27.6527813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:27.6528474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:27.6529768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:27.6530405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:27.7324041Z dist init r=1, world=4 2022-08-17T12:16:27.7324401Z dist init r=3, world=4 2022-08-17T12:16:27.7460750Z dist init r=0, world=4 2022-08-17T12:16:27.8097380Z dist init r=2, world=4 2022-08-17T12:16:27.8239705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:27.8341238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:27.8341998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:27.8342465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:27.8343171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:27.8343714Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:27.8344261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:27.8344817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:27.8449517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:27.8450054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:27.8450626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:27.8451212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:28.1824526Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:28.1881311Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8721 2022-08-17T12:16:28.1918956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8722 2022-08-17T12:16:28.1948301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8723 2022-08-17T12:16:28.1978246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8724 2022-08-17T12:16:29.2237886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:29.2238582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:29.2239420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:29.2240062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:29.2248788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:29.2249463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:29.2252042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:29.2252658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:29.2268571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:29.2269177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:29.2272115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:29.2272739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:29.2318454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:29.2319104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:29.2322084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:29.2322705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:29.3862696Z dist init r=2, world=4 2022-08-17T12:16:29.3866629Z dist init r=0, world=4 2022-08-17T12:16:29.3880686Z dist init r=1, world=4 2022-08-17T12:16:29.3918142Z dist init r=3, world=4 2022-08-17T12:16:29.4071943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:29.4173200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:29.4275492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:29.4276157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:29.4277130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:29.4277842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:29.4278676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:29.4279565Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:29.4282661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:29.4283209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:29.4283767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:29.4284307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:29.8017557Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:29.8075273Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8777 2022-08-17T12:16:29.8113058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8778 2022-08-17T12:16:29.8141485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8779 2022-08-17T12:16:29.8171460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8780 2022-08-17T12:16:30.8329241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:30.8329878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:30.8332469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:30.8333412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:30.8373915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:30.8374572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:30.8377618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:30.8378244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:30.8442925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:30.8443558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:30.8446335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:30.8446978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:30.8714769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:30.8715377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:30.8717235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:30.8717872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:30.9918219Z dist init r=2, world=4 2022-08-17T12:16:30.9976319Z dist init r=1, world=4 2022-08-17T12:16:31.0034069Z dist init r=0, world=4 2022-08-17T12:16:31.0278739Z dist init r=3, world=4 2022-08-17T12:16:31.0388767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:31.0432688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:31.0548978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:31.0549831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:31.0550491Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:31.0551130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:31.0551670Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:31.0592827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:31.0642550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:31.0643264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:31.0643823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:31.0644386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:31.4212531Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:31.4270525Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8833 2022-08-17T12:16:31.4309210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8834 2022-08-17T12:16:31.4340722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8835 2022-08-17T12:16:31.4372496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8836 2022-08-17T12:16:32.4753490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:32.4754794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:32.4755890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:32.4756536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:32.4849099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:32.4849706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:32.4851446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:32.4852054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:32.4911007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:32.4911701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:32.4913941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:32.4914834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:32.4964207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:32.4964813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:32.4967005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:32.4967636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:32.6364511Z dist init r=0, world=4 2022-08-17T12:16:32.6453520Z dist init r=2, world=4 2022-08-17T12:16:32.6519110Z dist init r=3, world=4 2022-08-17T12:16:32.6539940Z dist init r=1, world=4 2022-08-17T12:16:32.6731151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:32.6832482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:32.6933489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:32.6934024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:32.6934924Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:32.6936013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:32.6936711Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:32.6937255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:32.7042092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:32.7042826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:32.7043380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:32.7043919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:33.0413085Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:33.0469568Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8889 2022-08-17T12:16:33.0506782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8890 2022-08-17T12:16:33.0538141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8891 2022-08-17T12:16:33.0568027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8892 2022-08-17T12:16:34.0444002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:34.0444670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:34.0445748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:34.0446634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:34.0749225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:34.0749783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:34.0750711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:34.0751313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:34.0771632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:34.0772318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:34.0773995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:34.0774630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:34.1001754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:34.1002370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:34.1003742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:34.1004396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:34.2037775Z dist init r=0, world=4 2022-08-17T12:16:34.2368351Z dist init r=1, world=4 2022-08-17T12:16:34.2381505Z dist init r=3, world=4 2022-08-17T12:16:34.2592359Z dist init r=2, world=4 2022-08-17T12:16:34.2794289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:34.2894432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:34.2906199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:34.2906706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:34.2907414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:34.2907996Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:34.2998330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:34.2998937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:34.3104273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:34.3104862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:34.3105464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:34.3106060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:34.6607461Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:34.6607772Z 2022-08-17T12:16:34.6608264Z ---------------------------------------------------------------------- 2022-08-17T12:16:34.6608725Z Ran 8 tests in 13.217s 2022-08-17T12:16:34.6608940Z 2022-08-17T12:16:34.6609080Z OK (skipped=8) 2022-08-17T12:16:34.6609278Z 2022-08-17T12:16:34.6609451Z Generating XML reports... 2022-08-17T12:16:34.6655166Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220817121621.xml 2022-08-17T12:16:34.9828238Z Running distributed/fsdp/test_fsdp_fx ... [2022-08-17 12:16:34.982386] 2022-08-17T12:16:34.9828869Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:16:34.982471] 2022-08-17T12:16:36.1242987Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-08-17T12:16:36.1253656Z 2022-08-17T12:16:36.1253800Z Running tests... 2022-08-17T12:16:36.1254435Z ---------------------------------------------------------------------- 2022-08-17T12:16:36.1277439Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-08-17T12:16:36.3960879Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8956 2022-08-17T12:16:36.3989545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8957 2022-08-17T12:16:36.4018729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8958 2022-08-17T12:16:36.4049449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8959 2022-08-17T12:16:37.5014064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:37.5014711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:37.5016484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:37.5017164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:37.5317820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:37.5318376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:37.5319721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:37.5320234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:37.5492478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:37.5493147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:37.5494379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:37.5495045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:37.5603252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:37.5603940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:37.5605656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:37.5606584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:37.6602888Z dist init r=0, world=4 2022-08-17T12:16:37.6899770Z dist init r=3, world=4 2022-08-17T12:16:37.7065615Z dist init r=2, world=4 2022-08-17T12:16:37.7181398Z dist init r=1, world=4 2022-08-17T12:16:37.7477185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:37.7576591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:37.7679262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:37.7679952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:37.7681084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:37.7681784Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:37.7682335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:37.7682881Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:37.7786434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:37.7787137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:37.7787962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:37.7788789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:38.2096879Z ok (2.084s) 2022-08-17T12:16:38.2097133Z 2022-08-17T12:16:38.2097589Z ---------------------------------------------------------------------- 2022-08-17T12:16:38.2097987Z Ran 1 test in 2.084s 2022-08-17T12:16:38.2098187Z 2022-08-17T12:16:38.2098296Z OK 2022-08-17T12:16:38.2098448Z 2022-08-17T12:16:38.2098592Z Generating XML reports... 2022-08-17T12:16:38.2134854Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220817121636.xml 2022-08-17T12:16:38.5310316Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-08-17 12:16:38.530586] 2022-08-17T12:16:38.5310998Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:16:38.530677] 2022-08-17T12:16:39.6698296Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc 2022-08-17T12:16:39.6711057Z 2022-08-17T12:16:39.6711751Z Running tests... 2022-08-17T12:16:39.6712277Z ---------------------------------------------------------------------- 2022-08-17T12:16:39.6723661Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-08-17T12:16:39.9339942Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9023 2022-08-17T12:16:39.9368842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9024 2022-08-17T12:16:39.9397346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9025 2022-08-17T12:16:39.9428598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9026 2022-08-17T12:16:41.0048284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:41.0048921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:41.0049929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:41.0050535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:41.0081918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:41.0082595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:41.0086121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:41.0087081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:41.0091380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:41.0092015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:41.0095576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:41.0096241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:41.0406937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:41.0407593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:41.0408663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:41.0409312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:41.1686968Z dist init r=3, world=4 2022-08-17T12:16:41.1731526Z dist init r=1, world=4 2022-08-17T12:16:41.1827654Z dist init r=2, world=4 2022-08-17T12:16:41.2014737Z dist init r=0, world=4 2022-08-17T12:16:41.2427350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:41.2528508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:41.2629784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:41.2630951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:41.2632023Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:41.2632903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:41.2633470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:41.2634020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:41.2737503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:41.2738216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:41.2738762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:41.2739344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:41.6475217Z skip: Need at least 2 CUDA devices (1.976s) 2022-08-17T12:16:41.6487584Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-08-17T12:16:41.6535227Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9079 2022-08-17T12:16:41.6573049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9080 2022-08-17T12:16:41.6602391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9081 2022-08-17T12:16:41.6631740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9082 2022-08-17T12:16:42.6771286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:42.6771936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:42.6773288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:42.6774168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:42.6922915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:42.6923587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:42.6925317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:42.6926233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:42.7031852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:42.7032496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:42.7033958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:42.7034596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:42.7170913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:42.7171429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:42.7173429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:42.7173993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:42.8381105Z dist init r=3, world=4 2022-08-17T12:16:42.8529831Z dist init r=1, world=4 2022-08-17T12:16:42.8623339Z dist init r=2, world=4 2022-08-17T12:16:42.8773361Z dist init r=0, world=4 2022-08-17T12:16:42.9185938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:42.9186719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:42.9187720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:42.9188874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:42.9189487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:42.9190034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:42.9190570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:42.9191197Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:42.9193978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:42.9194673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:42.9210985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:42.9211402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:43.2672036Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:43.2683772Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc) 2022-08-17T12:16:43.2730811Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9135 2022-08-17T12:16:43.2768847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9136 2022-08-17T12:16:43.2799248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9137 2022-08-17T12:16:43.2829816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9138 2022-08-17T12:16:44.2760692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:44.2761348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:44.2762936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:44.2763528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:44.3056791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:44.3057446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:44.3058652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:44.3059241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:44.3227729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:44.3228406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:44.3229974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:44.3230629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:44.3413757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:44.3414381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:44.3415863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:44.3416495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:44.4342041Z dist init r=3, world=4 2022-08-17T12:16:44.4642096Z dist init r=1, world=4 2022-08-17T12:16:44.4811691Z dist init r=0, world=4 2022-08-17T12:16:44.4988765Z dist init r=2, world=4 2022-08-17T12:16:44.5152701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:44.5253460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:44.5300802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:44.5301239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:44.5301895Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:44.5302439Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:44.5355416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:44.5356158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:44.5407984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:44.5408551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:44.5409120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:44.5409723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:44.8870034Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:44.8881623Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-08-17T12:16:44.8928823Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9191 2022-08-17T12:16:44.8970065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9192 2022-08-17T12:16:44.8999397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9193 2022-08-17T12:16:44.9028290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9194 2022-08-17T12:16:45.9154036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:45.9154663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:45.9156197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:45.9156848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:45.9175913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:45.9176540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:45.9178725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:45.9179375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:45.9414766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:45.9415384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:45.9416889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:45.9417528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:45.9418329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:45.9418855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:45.9423232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:45.9423866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:46.0751141Z dist init r=3, world=4 2022-08-17T12:16:46.0772720Z dist init r=0, world=4 2022-08-17T12:16:46.1014004Z dist init r=1, world=4 2022-08-17T12:16:46.1014245Z dist init r=2, world=4 2022-08-17T12:16:46.1263403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:46.1363472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:46.1364098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:46.1364858Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:46.1365360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:46.1366213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:46.1367253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:46.1367834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:46.1472950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:46.1473574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:46.1474144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:46.1474499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:46.5068179Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:46.5079544Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-08-17T12:16:46.5129764Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9247 2022-08-17T12:16:46.5167618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9248 2022-08-17T12:16:46.5198301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9249 2022-08-17T12:16:46.5231433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9250 2022-08-17T12:16:47.5037357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:47.5038072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:47.5039164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:47.5039860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:47.5200246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:47.5200961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:47.5202228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:47.5202899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:47.5699560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:47.5700248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:47.5701600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:47.5702227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:47.5854172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:47.5854820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:47.5856245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:47.5856862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:47.6624250Z dist init r=0, world=4 2022-08-17T12:16:47.6789146Z dist init r=1, world=4 2022-08-17T12:16:47.7287841Z dist init r=2, world=4 2022-08-17T12:16:47.7427894Z dist init r=3, world=4 2022-08-17T12:16:47.7538096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:47.7639609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:47.7640062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:47.7640460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:47.7641345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:47.7642430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:47.7643004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:47.7643732Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:47.7748210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:47.7748996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:47.7749440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:47.7749803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:48.1269478Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:48.1281611Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc) 2022-08-17T12:16:48.1330260Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9303 2022-08-17T12:16:48.1368690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9304 2022-08-17T12:16:48.1399248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9305 2022-08-17T12:16:48.1428698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9306 2022-08-17T12:16:49.1652429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:49.1653123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:49.1654797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:49.1655436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:49.1750229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:49.1750965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:49.1753481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:49.1754178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:49.1863855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:49.1864508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:49.1867512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:49.1868101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:49.2074968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:49.2075704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:49.2076688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:49.2077395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:49.3257819Z dist init r=2, world=4 2022-08-17T12:16:49.3361089Z dist init r=0, world=4 2022-08-17T12:16:49.3456433Z dist init r=1, world=4 2022-08-17T12:16:49.3621675Z dist init r=3, world=4 2022-08-17T12:16:49.3729712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:49.3830975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:49.3933501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:49.3934174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:49.3935265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:49.3936025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:49.3936712Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:49.3937558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:49.3941111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:49.3941678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:49.3942235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:49.3943473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:49.7468937Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:49.7481455Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-08-17T12:16:49.7531324Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9359 2022-08-17T12:16:49.7570163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9360 2022-08-17T12:16:49.7599375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9361 2022-08-17T12:16:49.7628917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9362 2022-08-17T12:16:50.7810990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:50.7811693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:50.7812979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:50.7813675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:50.7930520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:50.7931115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:50.7933318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:50.7933961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:50.7941299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:50.7941878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:50.7944896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:50.7945537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:50.8018915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:50.8019565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:50.8022222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:50.8022890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:50.9411066Z dist init r=3, world=4 2022-08-17T12:16:50.9536459Z dist init r=0, world=4 2022-08-17T12:16:50.9541293Z dist init r=2, world=4 2022-08-17T12:16:50.9593750Z dist init r=1, world=4 2022-08-17T12:16:50.9720748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:50.9903958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:51.0006284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:51.0007194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:51.0007631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:51.0008158Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:51.0009978Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:51.0024284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:51.0114793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:51.0115565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:51.0116178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:51.0116629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:51.3668866Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:51.3680402Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-08-17T12:16:51.3729567Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9415 2022-08-17T12:16:51.3766946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9416 2022-08-17T12:16:51.3796767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9417 2022-08-17T12:16:51.3826422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9418 2022-08-17T12:16:52.4414245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:52.4414894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:52.4416141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:52.4416793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:52.4522166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:52.4524495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:52.4525365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:52.4526289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:52.4712293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:52.4712880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:52.4714506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:52.4715103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:52.4715923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:52.4716528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:52.4717699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:52.4718256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:52.6026135Z dist init r=0, world=4 2022-08-17T12:16:52.6127613Z dist init r=3, world=4 2022-08-17T12:16:52.6303110Z dist init r=1, world=4 2022-08-17T12:16:52.6311948Z dist init r=2, world=4 2022-08-17T12:16:52.6527792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:52.6623730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:52.6726596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:52.6727364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:52.6727877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:52.6728455Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:52.6729192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:52.6730479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:52.6834361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:52.6834980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:52.6835600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:52.6836144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:52.9867197Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:52.9878809Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc) 2022-08-17T12:16:52.9926377Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9471 2022-08-17T12:16:52.9963531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9472 2022-08-17T12:16:52.9992442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9473 2022-08-17T12:16:53.0022111Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9474 2022-08-17T12:16:54.0098030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:54.0098730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:54.0099602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:54.0100242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:54.0101045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:54.0101701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:54.0102527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:54.0103147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:54.0170943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:54.0171574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:54.0174589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:54.0175272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:54.0694853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:54.0695608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:54.0696685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:54.0697405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:54.1721602Z dist init r=0, world=4 2022-08-17T12:16:54.1721959Z dist init r=2, world=4 2022-08-17T12:16:54.1768792Z dist init r=3, world=4 2022-08-17T12:16:54.2300927Z dist init r=1, world=4 2022-08-17T12:16:54.2535626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:54.2613906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:54.2614685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:54.2615299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:54.2615687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:54.2616224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:54.2616765Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:54.2637595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:54.2723396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:54.2724027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:54.2724580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:54.2725146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:54.6062846Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:16:54.6074743Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-08-17T12:16:54.6122333Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9527 2022-08-17T12:16:54.6160976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9528 2022-08-17T12:16:54.6191826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9529 2022-08-17T12:16:54.6223064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9530 2022-08-17T12:16:55.6486274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:55.6487023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:55.6487895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:55.6488459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:55.6773582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:55.6774223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:55.6776241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:55.6776908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:55.6797670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:55.6798323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:55.6801220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:55.6801897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:55.6970690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:55.6971341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:55.6972938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:55.6973594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:55.8100664Z dist init r=0, world=4 2022-08-17T12:16:55.8439363Z dist init r=1, world=4 2022-08-17T12:16:55.8578496Z dist init r=2, world=4 2022-08-17T12:16:55.8584179Z dist init r=3, world=4 2022-08-17T12:16:55.8788153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:55.8851447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:55.8852562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:55.8853314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:55.8853981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:55.8854656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:55.8855211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:55.8890655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:55.8960376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:55.8961002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:55.8961581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:55.8962099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:56.2263072Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:56.2275064Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-08-17T12:16:56.2324034Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9583 2022-08-17T12:16:56.2363395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9584 2022-08-17T12:16:56.2394546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9585 2022-08-17T12:16:56.2424542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9586 2022-08-17T12:16:57.2708372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:57.2708954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:57.2710385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:57.2711121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:57.2865769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:57.2866407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:57.2867377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:57.2868014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:57.3381493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:57.3382900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:57.3383743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:57.3384390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:57.3706341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:57.3707235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:57.3707958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:57.3708485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:57.4335261Z dist init r=2, world=4 2022-08-17T12:16:57.4484619Z dist init r=3, world=4 2022-08-17T12:16:57.4975260Z dist init r=1, world=4 2022-08-17T12:16:57.5279127Z dist init r=0, world=4 2022-08-17T12:16:57.5450312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:57.5652771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:57.5754433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:57.5757146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:57.5757812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:57.5758355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:57.5758906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:57.5759453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:57.5763035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:57.5763728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:57.5764409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:57.5764872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:57.9466141Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:16:57.9478008Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc) 2022-08-17T12:16:57.9525367Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9639 2022-08-17T12:16:57.9565057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9640 2022-08-17T12:16:57.9596231Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9641 2022-08-17T12:16:57.9626315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9642 2022-08-17T12:16:58.9501441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:58.9502086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:58.9504450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:58.9505072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:58.9631093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:58.9631773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:58.9633933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:58.9634534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:58.9696861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:58.9697697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:58.9699992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:58.9700634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:59.0144359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:16:59.0144929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:16:59.0146834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:16:59.0147395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:16:59.1079969Z dist init r=3, world=4 2022-08-17T12:16:59.1233473Z dist init r=0, world=4 2022-08-17T12:16:59.1277974Z dist init r=1, world=4 2022-08-17T12:16:59.1700931Z dist init r=2, world=4 2022-08-17T12:16:59.1891588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:16:59.2011584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:16:59.2012033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:16:59.2012971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:16:59.2013685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:59.2014256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:59.2014808Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:59.2095072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:16:59.2119213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:16:59.2119676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:16:59.2120229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:16:59.2120784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:16:59.5667595Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:16:59.5667837Z 2022-08-17T12:16:59.5668366Z ---------------------------------------------------------------------- 2022-08-17T12:16:59.5668951Z Ran 12 tests in 19.896s 2022-08-17T12:16:59.5669077Z 2022-08-17T12:16:59.5669140Z OK (skipped=12) 2022-08-17T12:16:59.5669256Z 2022-08-17T12:16:59.5669347Z Generating XML reports... 2022-08-17T12:16:59.5716795Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220817121639.xml 2022-08-17T12:16:59.8919106Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-08-17 12:16:59.891502] 2022-08-17T12:16:59.8919937Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:16:59.891589] 2022-08-17T12:17:01.0247247Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules 2022-08-17T12:17:01.0260436Z 2022-08-17T12:17:01.0260699Z Running tests... 2022-08-17T12:17:01.0261138Z ---------------------------------------------------------------------- 2022-08-17T12:17:01.0272616Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules) 2022-08-17T12:17:01.2923831Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9706 2022-08-17T12:17:01.2954390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9707 2022-08-17T12:17:01.2983990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9708 2022-08-17T12:17:01.3016126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9709 2022-08-17T12:17:02.3047160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:02.3047776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:02.3049158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:02.3049818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:02.3055715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:02.3056389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:02.3060103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:02.3060844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:02.3107384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:02.3108011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:02.3110574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:02.3111295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:02.3733303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:02.3733972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:02.3736507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:02.3737154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:02.4643238Z dist init r=3, world=4 2022-08-17T12:17:02.4643648Z dist init r=2, world=4 2022-08-17T12:17:02.4689940Z dist init r=0, world=4 2022-08-17T12:17:02.5261479Z dist init r=1, world=4 2022-08-17T12:17:02.5574492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:02.5676290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:17:02.5677218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:02.5677622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:17:02.5678264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:02.5678821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:02.5679358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:02.5679905Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:02.5783612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:02.5784199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:17:02.5784805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:17:02.5785307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:02.9059769Z skip: Need at least 2 CUDA devices (1.880s) 2022-08-17T12:17:02.9069899Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules) 2022-08-17T12:17:02.9117555Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9762 2022-08-17T12:17:02.9154791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9763 2022-08-17T12:17:02.9183601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9764 2022-08-17T12:17:02.9213273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9765 2022-08-17T12:17:03.9560337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:03.9561003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:03.9562324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:03.9562940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:03.9702254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:03.9702883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:03.9703973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:03.9704581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:03.9813460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:03.9814140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:03.9815137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:03.9815754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:04.0069186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:04.0069813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:04.0071429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:04.0072049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:04.1173135Z dist init r=1, world=4 2022-08-17T12:17:04.1346775Z dist init r=0, world=4 2022-08-17T12:17:04.1608344Z dist init r=2, world=4 2022-08-17T12:17:04.1710252Z dist init r=3, world=4 2022-08-17T12:17:04.1920797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:04.1922106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:17:04.1923184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:04.1923847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:04.1924667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:04.1925172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:17:04.1925710Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:04.1926508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:04.1933065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:17:04.1933825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:04.1934390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:04.1934938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:17:04.5254395Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:17:04.5261228Z test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules) 2022-08-17T12:17:04.5309100Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9818 2022-08-17T12:17:04.5346710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9819 2022-08-17T12:17:04.5375640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9820 2022-08-17T12:17:04.5404147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9821 2022-08-17T12:17:05.5380449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:05.5381108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:05.5382159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:05.5382811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:05.5579170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:05.5579846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:05.5581677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:05.5582356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:05.5839492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:05.5840175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:05.5841555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:05.5842185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:05.5961746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:05.5962316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:05.5964497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:05.5965039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:05.6963708Z dist init r=1, world=4 2022-08-17T12:17:05.7161649Z dist init r=2, world=4 2022-08-17T12:17:05.7414471Z dist init r=3, world=4 2022-08-17T12:17:05.7521581Z dist init r=0, world=4 2022-08-17T12:17:05.7775836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:05.7878415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:17:05.7879415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:05.7880009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:05.7880812Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:05.7881379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:17:05.7882243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:05.7883079Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:05.7887045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:05.7887655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:17:05.7888239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:17:05.7888763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:06.1445154Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:17:06.1455387Z test_ignored_modules_nested (__main__.TestFSDPIgnoredModules) 2022-08-17T12:17:06.1504477Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9874 2022-08-17T12:17:06.1542752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9875 2022-08-17T12:17:06.1572398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9876 2022-08-17T12:17:06.1602140Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9877 2022-08-17T12:17:07.2116194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:07.2116900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:07.2118608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:07.2119281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:07.2137451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:07.2138142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:07.2141068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:07.2141657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:07.2202871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:07.2203546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:07.2206257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:07.2207171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:07.2347261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:07.2347996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:07.2350531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:07.2351354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:07.3725302Z dist init r=1, world=4 2022-08-17T12:17:07.3757444Z dist init r=0, world=4 2022-08-17T12:17:07.3810370Z dist init r=2, world=4 2022-08-17T12:17:07.4021002Z dist init r=3, world=4 2022-08-17T12:17:07.4136323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:07.4225572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:07.4226003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:17:07.4226459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:17:07.4227329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:07.4227968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:07.4228518Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:07.4238868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:07.4332950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:17:07.4333584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:07.4334134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:07.4334691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:17:07.7642882Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:17:07.7654112Z test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules) 2022-08-17T12:17:07.7703446Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9930 2022-08-17T12:17:07.7741336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9931 2022-08-17T12:17:07.7770355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9932 2022-08-17T12:17:07.7799495Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9933 2022-08-17T12:17:08.7733075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:08.7733722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:08.7735549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:08.7736206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:08.8223359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:08.8224036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:08.8225494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:08.8226129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:08.8226955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:08.8227842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:08.8229881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:08.8230534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:08.8283056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:08.8283750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:08.8286689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:08.8287336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:08.9343673Z dist init r=2, world=4 2022-08-17T12:17:08.9825416Z dist init r=1, world=4 2022-08-17T12:17:08.9842576Z dist init r=0, world=4 2022-08-17T12:17:08.9867461Z dist init r=3, world=4 2022-08-17T12:17:08.9977543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:09.0079075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:09.0181358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:17:09.0182422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:09.0183175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:17:09.0204459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:09.0205688Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:09.0206905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:17:09.0246675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:09.0247393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:17:09.0248116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:09.0248815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:17:09.2837413Z skip: Need at least 2 CUDA devices (1.519s) 2022-08-17T12:17:09.2837757Z 2022-08-17T12:17:09.2838251Z ---------------------------------------------------------------------- 2022-08-17T12:17:09.2838537Z Ran 5 tests in 8.258s 2022-08-17T12:17:09.2838664Z 2022-08-17T12:17:09.2838741Z OK (skipped=5) 2022-08-17T12:17:09.2838854Z 2022-08-17T12:17:09.2838944Z Generating XML reports... 2022-08-17T12:17:09.2879688Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220817121701.xml 2022-08-17T12:17:09.6219842Z Running distributed/fsdp/test_fsdp_input ... [2022-08-17 12:17:09.621560] 2022-08-17T12:17:09.6220452Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:17:09.621649] 2022-08-17T12:17:10.7464929Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2022-08-17T12:17:10.7476573Z 2022-08-17T12:17:10.7476732Z Running tests... 2022-08-17T12:17:10.7477164Z ---------------------------------------------------------------------- 2022-08-17T12:17:10.7490591Z test_input_type_dict (__main__.TestInput) 2022-08-17T12:17:11.0102661Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9997 2022-08-17T12:17:11.9671730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:11.9672204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:11.9673456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:11.9673895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:12.1160852Z dist init r=0, world=1 2022-08-17T12:17:12.1169389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:12.1171985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:17:12.1175848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:12.4133910Z skip: Need at least 1 CUDA device (1.665s) 2022-08-17T12:17:12.4146730Z test_input_type_list (__main__.TestInput) 2022-08-17T12:17:12.4198708Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10011 2022-08-17T12:17:13.3646897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:13.3647302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:13.3648496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:13.3648869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:13.5144157Z dist init r=0, world=1 2022-08-17T12:17:13.5152308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:13.5154806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:17:13.5159183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:13.8227154Z skip: Need at least 1 CUDA device (1.409s) 2022-08-17T12:17:13.8227416Z 2022-08-17T12:17:13.8227937Z ---------------------------------------------------------------------- 2022-08-17T12:17:13.8228387Z Ran 2 tests in 3.075s 2022-08-17T12:17:13.8228520Z 2022-08-17T12:17:13.8228584Z OK (skipped=2) 2022-08-17T12:17:13.8228700Z 2022-08-17T12:17:13.8228792Z Generating XML reports... 2022-08-17T12:17:13.8265855Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220817121710.xml 2022-08-17T12:17:14.1423959Z Running distributed/fsdp/test_fsdp_memory ... [2022-08-17 12:17:14.142004] 2022-08-17T12:17:14.1424563Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:17:14.142091] 2022-08-17T12:17:15.2768221Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-08-17T12:17:15.2780258Z 2022-08-17T12:17:15.2780348Z Running tests... 2022-08-17T12:17:15.2781316Z ---------------------------------------------------------------------- 2022-08-17T12:17:15.5473408Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10036 2022-08-17T12:17:15.5499792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10037 2022-08-17T12:17:16.5301860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:16.5302499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:16.5303542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:16.5304394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:16.5496960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:16.5497601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:16.5499157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:16.5499792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:16.6848193Z dist init r=0, world=2 2022-08-17T12:17:16.7049439Z dist init r=1, world=2 2022-08-17T12:17:16.7157895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:16.7158996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:16.7160656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:16.7162670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:16.7267001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:16.7267618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:17.0534403Z skip: Need at least 2 CUDA devices (1.775s) 2022-08-17T12:17:17.0600849Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10064 2022-08-17T12:17:17.0637733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10065 2022-08-17T12:17:18.0267223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:18.0267660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:18.0268749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:18.0269166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:18.0325349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:18.0325699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:18.0328895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:18.0329291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:18.1786067Z dist init r=1, world=2 2022-08-17T12:17:18.1858333Z dist init r=0, world=2 2022-08-17T12:17:18.1994516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:18.1994944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:18.1997726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:18.1998565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:18.2002406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:18.2002875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:18.4668820Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:17:18.4669089Z 2022-08-17T12:17:18.4669543Z ---------------------------------------------------------------------- 2022-08-17T12:17:18.4669969Z Ran 2 tests in 3.189s 2022-08-17T12:17:18.4670154Z 2022-08-17T12:17:18.4670270Z OK (skipped=2) 2022-08-17T12:17:18.4670445Z 2022-08-17T12:17:18.4670972Z Generating XML reports... 2022-08-17T12:17:18.4706389Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220817121715.xml 2022-08-17T12:17:18.8003807Z Running distributed/fsdp/test_fsdp_meta ... [2022-08-17 12:17:18.799995] 2022-08-17T12:17:18.8004399Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:17:18.800078] 2022-08-17T12:17:19.9322704Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2022-08-17T12:17:19.9336182Z 2022-08-17T12:17:19.9336634Z Running tests... 2022-08-17T12:17:19.9337060Z ---------------------------------------------------------------------- 2022-08-17T12:17:20.1948465Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10103 2022-08-17T12:17:20.1976773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10104 2022-08-17T12:17:21.1629522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:21.1630021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:21.1631628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:21.1632013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:21.1939353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:21.1939948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:21.1941856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:21.1942240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:21.3170916Z dist init r=0, world=2 2022-08-17T12:17:21.3479579Z dist init r=1, world=2 2022-08-17T12:17:21.3682558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:21.3683069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:21.3683867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:21.3684495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:21.3790542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:21.3791004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:21.7036172Z skip: Need at least 2 CUDA devices (1.770s) 2022-08-17T12:17:21.7042748Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-08-17T12:17:21.7095145Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10131 2022-08-17T12:17:21.7133204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10132 2022-08-17T12:17:22.6885529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:22.6886420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:22.6887488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:22.6888135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:22.7155261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:22.7156198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:22.7157340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:22.7157959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:22.8425521Z dist init r=0, world=2 2022-08-17T12:17:22.8692927Z dist init r=1, world=2 2022-08-17T12:17:22.8836099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:22.8836608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:22.8837261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:22.8837806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:22.8943741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:22.8944139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:23.2164816Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:23.2220734Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10159 2022-08-17T12:17:23.2259186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10160 2022-08-17T12:17:24.1993771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:24.1994317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:24.1995594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:24.1996189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:24.1997813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:24.1998340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:24.2000748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:24.2001329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:24.3542370Z dist init r=0, world=2 2022-08-17T12:17:24.3550843Z dist init r=1, world=2 2022-08-17T12:17:24.3659642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:24.3660329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:24.3660966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:24.3661538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:24.3767064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:24.3767568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:24.6289140Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:17:24.6342390Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10187 2022-08-17T12:17:24.6379803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10188 2022-08-17T12:17:25.6091093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:25.6092024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:25.6093108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:25.6093753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:25.6105246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:25.6105910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:25.6108891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:25.6109499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:25.7641241Z dist init r=1, world=2 2022-08-17T12:17:25.7663885Z dist init r=0, world=2 2022-08-17T12:17:25.7850596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:25.7851253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:25.7852016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:25.7852732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:25.7857172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:25.7857556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:26.0411289Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:17:26.0463661Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10215 2022-08-17T12:17:26.0501830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10216 2022-08-17T12:17:27.0483214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:27.0483694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:27.0485337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:27.0485714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:27.0710805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:27.0711176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:27.0713911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:27.0714329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:27.2045208Z dist init r=0, world=2 2022-08-17T12:17:27.2242840Z dist init r=1, world=2 2022-08-17T12:17:27.2351560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:27.2352064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:27.2352692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:27.2353250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:27.2459359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:27.2459934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:27.5535483Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:27.5542158Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-08-17T12:17:27.5546738Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-08-17T12:17:27.5552187Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-08-17T12:17:27.5556667Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-08-17T12:17:27.5607438Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10243 2022-08-17T12:17:27.5645343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10244 2022-08-17T12:17:28.5574195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:28.5574868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:28.5576319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:28.5576936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:28.5609351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:28.5610038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:28.5612872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:28.5613476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:28.7109793Z dist init r=1, world=2 2022-08-17T12:17:28.7152452Z dist init r=0, world=2 2022-08-17T12:17:28.7318960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:28.7319428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:28.7320157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:28.7320738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:28.7327258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:28.7327805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:29.0681222Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:29.0734568Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10271 2022-08-17T12:17:29.0771563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10272 2022-08-17T12:17:30.0556740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:30.0557126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:30.0559178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:30.0559554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:30.0822502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:30.0822868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:30.0825165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:30.0825536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:30.2092352Z dist init r=1, world=2 2022-08-17T12:17:30.2343216Z dist init r=0, world=2 2022-08-17T12:17:30.2501896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:30.2502450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:30.2503277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:30.2503948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:30.2510268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:30.2510812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:30.5803145Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:30.5809353Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-08-17T12:17:30.5813590Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-08-17T12:17:30.5814297Z 2022-08-17T12:17:30.5814742Z ---------------------------------------------------------------------- 2022-08-17T12:17:30.5815207Z Ran 14 tests in 10.648s 2022-08-17T12:17:30.5815427Z 2022-08-17T12:17:30.5815545Z OK (skipped=14) 2022-08-17T12:17:30.5815744Z 2022-08-17T12:17:30.5815842Z Generating XML reports... 2022-08-17T12:17:30.5861899Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220817121719.xml 2022-08-17T12:17:30.9029872Z Running distributed/fsdp/test_fsdp_misc ... [2022-08-17 12:17:30.902589] 2022-08-17T12:17:30.9030499Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:17:30.902677] 2022-08-17T12:17:32.0363424Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc 2022-08-17T12:17:32.0375791Z 2022-08-17T12:17:32.0375912Z Running tests... 2022-08-17T12:17:32.0376365Z ---------------------------------------------------------------------- 2022-08-17T12:17:32.0383980Z test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc) 2022-08-17T12:17:32.3029093Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10310 2022-08-17T12:17:32.3057843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10311 2022-08-17T12:17:33.2876700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:33.2877347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:33.2879055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:33.2879698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:33.2985058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:33.2985718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:33.2987931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:33.2988564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:33.4440376Z dist init r=0, world=2 2022-08-17T12:17:33.4537741Z dist init r=1, world=2 2022-08-17T12:17:33.4649599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:33.4650382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:33.4651122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:33.4651850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:33.4655480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:33.4655901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:33.7092083Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T12:17:33.7100439Z test_device_id_auto_wrap (__main__.TestFSDPMisc) 2022-08-17T12:17:33.7147466Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10338 2022-08-17T12:17:33.7183042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10339 2022-08-17T12:17:34.6916588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:34.6916993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:34.6919263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:34.6919732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:34.7299994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:34.7300596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:34.7301676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:34.7302029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:34.8457118Z dist init r=1, world=2 2022-08-17T12:17:34.8829600Z dist init r=0, world=2 2022-08-17T12:17:34.9039853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:34.9040504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:34.9041247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:34.9041818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:34.9045301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:34.9045725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:35.2214816Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:35.2224894Z test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc) 2022-08-17T12:17:35.2273206Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10366 2022-08-17T12:17:35.2310635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10367 2022-08-17T12:17:36.2152969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:36.2153357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:36.2153812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:36.2154554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:36.2155295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:36.2155657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:36.2156211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:36.2156574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:36.3708518Z dist init r=1, world=2 2022-08-17T12:17:36.3723981Z dist init r=0, world=2 2022-08-17T12:17:36.3934141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:36.3934856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:36.3935750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:36.3936313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:36.4040606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:36.4041369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:36.7343274Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:17:36.7353142Z test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc) 2022-08-17T12:17:36.7403980Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10394 2022-08-17T12:17:36.7442881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10395 2022-08-17T12:17:37.7407018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:37.7407712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:37.7408835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:37.7409439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:37.7454643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:37.7455042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:37.7457610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:37.7458088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:37.8951342Z dist init r=0, world=2 2022-08-17T12:17:37.9119939Z dist init r=1, world=2 2022-08-17T12:17:37.9261133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:37.9261804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:37.9262449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:37.9263003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:37.9368637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:37.9369263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:38.2475410Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:17:38.2492527Z test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc) 2022-08-17T12:17:38.2542215Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10422 2022-08-17T12:17:38.2580389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10423 2022-08-17T12:17:39.2279537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:39.2279975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:39.2281129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:39.2281765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:39.2316549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:39.2317069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:39.2319839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:39.2320352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:39.3814621Z dist init r=1, world=2 2022-08-17T12:17:39.3850261Z dist init r=0, world=2 2022-08-17T12:17:39.4023372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:39.4024102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:39.4024766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:39.4025328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:39.4129497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:39.4129911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:39.6610956Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:17:39.6628060Z test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc) 2022-08-17T12:17:39.6676586Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10450 2022-08-17T12:17:39.6712951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10451 2022-08-17T12:17:40.6325627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:40.6326516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:40.6327370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:40.6328028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:40.6338678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:40.6339299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:40.6341690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:40.6342385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:40.7861471Z dist init r=1, world=2 2022-08-17T12:17:40.7865573Z dist init r=0, world=2 2022-08-17T12:17:40.8070850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:40.8071430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:40.8072085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:40.8072668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:40.8077335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:40.8077732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:41.0742817Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:17:41.0804234Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10478 2022-08-17T12:17:41.0841823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10479 2022-08-17T12:17:42.0811519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:42.0812067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:42.0813668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:42.0814057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:42.0955255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:42.0955634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:42.0958689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:42.0959104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:42.2358558Z dist init r=0, world=2 2022-08-17T12:17:42.2482438Z dist init r=1, world=2 2022-08-17T12:17:42.2591683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:42.2592793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:42.2593689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:42.2594560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:42.2598185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:42.2599857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:42.5875345Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:17:42.5937571Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10506 2022-08-17T12:17:42.5974159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10507 2022-08-17T12:17:43.5753582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:43.5754218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:43.5755428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:43.5756089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:43.5834282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:43.5837213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:43.5837891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:43.5838266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:43.7299625Z dist init r=0, world=2 2022-08-17T12:17:43.7390070Z dist init r=1, world=2 2022-08-17T12:17:43.7499773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:43.7500467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:43.7501136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:43.7501702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:43.7606586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:43.7607056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:44.1007525Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:17:44.1071944Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10534 2022-08-17T12:17:44.1109619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10535 2022-08-17T12:17:45.1047425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:45.1047832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:45.1049865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:45.1050235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:45.1122133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:45.1122491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:45.1126524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:45.1126921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:45.2604824Z dist init r=0, world=2 2022-08-17T12:17:45.2664871Z dist init r=1, world=2 2022-08-17T12:17:45.2814380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:45.2815164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:45.2815807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:45.2816361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:45.2920450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:45.2921083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:45.6144339Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:17:45.6210037Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10562 2022-08-17T12:17:45.6246864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10563 2022-08-17T12:17:46.5992145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:46.5992840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:46.5994259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:46.5994922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:46.6125719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:46.6126576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:46.6128554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:46.6129190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:46.7530707Z dist init r=1, world=2 2022-08-17T12:17:46.7671221Z dist init r=0, world=2 2022-08-17T12:17:46.7840068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:46.7840499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:46.7841422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:46.7841991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:46.7846093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:46.7846711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:47.0277331Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:17:47.0335946Z test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10590 2022-08-17T12:17:47.0372202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10591 2022-08-17T12:17:48.0015855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:48.0016506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:48.0018061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:48.0018723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:48.0102172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:48.0102841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:48.0104904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:48.0105564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:48.1557865Z dist init r=0, world=2 2022-08-17T12:17:48.1627095Z dist init r=1, world=2 2022-08-17T12:17:48.1735751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:48.1736344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:48.1737125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:48.1737670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:48.1841561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:48.1841985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:48.4401228Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:17:48.4414558Z test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc) 2022-08-17T12:17:48.4469305Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10618 2022-08-17T12:17:48.4506015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10619 2022-08-17T12:17:49.4406710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:49.4407361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:49.4408422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:49.4408799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:49.4635672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:49.4636051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:49.4638403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:49.4638845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:49.6058265Z dist init r=1, world=2 2022-08-17T12:17:49.6195879Z dist init r=0, world=2 2022-08-17T12:17:49.6406073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:49.6406564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:49.6407324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:49.6408139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:49.6411601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:49.6412183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:49.9536821Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:17:49.9543847Z test_module_device_mismatches_device_id (__main__.TestFSDPMisc) 2022-08-17T12:17:49.9592709Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10646 2022-08-17T12:17:49.9628363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10647 2022-08-17T12:17:50.9609493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:50.9609892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:50.9612644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:50.9613013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:50.9711848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:50.9712213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:50.9715041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:50.9715407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:51.1156428Z dist init r=1, world=2 2022-08-17T12:17:51.1239501Z dist init r=0, world=2 2022-08-17T12:17:51.1448830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:51.1449454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:51.1450165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:51.1450804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:51.1455170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:51.1455814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:51.4659457Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:51.4666848Z test_multi_device_not_supported (__main__.TestFSDPMisc) 2022-08-17T12:17:51.4717510Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10674 2022-08-17T12:17:51.4756409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10675 2022-08-17T12:17:52.4518745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:52.4519213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:52.4520410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:52.4520921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:52.4589635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:52.4590108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:52.4592842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:52.4593391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:52.6049906Z dist init r=1, world=2 2022-08-17T12:17:52.6118056Z dist init r=0, world=2 2022-08-17T12:17:52.6259935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:52.6261054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:52.6261681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:52.6262216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:52.6267170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:52.6267644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:52.8786429Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:17:52.8794301Z test_no_params (__main__.TestFSDPMisc) 2022-08-17T12:17:52.8841690Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10702 2022-08-17T12:17:52.8879468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10703 2022-08-17T12:17:53.8750981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:53.8751630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:53.8753085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:53.8753755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:53.8840619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:53.8841280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:53.8843899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:53.8844305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:54.0308683Z dist init r=1, world=2 2022-08-17T12:17:54.0384875Z dist init r=0, world=2 2022-08-17T12:17:54.0516584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:54.0517077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:54.0517952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:54.0518696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:54.0622831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:54.0623320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:54.3911882Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:17:54.3912423Z 2022-08-17T12:17:54.3912966Z ---------------------------------------------------------------------- 2022-08-17T12:17:54.3913452Z Ran 15 tests in 22.354s 2022-08-17T12:17:54.3913681Z 2022-08-17T12:17:54.3913792Z OK (skipped=15) 2022-08-17T12:17:54.3913906Z 2022-08-17T12:17:54.3913995Z Generating XML reports... 2022-08-17T12:17:54.3963305Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220817121732.xml 2022-08-17T12:17:54.7161932Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-08-17 12:17:54.715668] 2022-08-17T12:17:54.7162564Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:17:54.715752] 2022-08-17T12:17:56.1055752Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision 2022-08-17T12:17:56.1072783Z 2022-08-17T12:17:56.1072917Z Running tests... 2022-08-17T12:17:56.1073575Z ---------------------------------------------------------------------- 2022-08-17T12:17:56.1260722Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10741 2022-08-17T12:17:56.1289964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10742 2022-08-17T12:17:57.0967381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:57.0967866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:57.0969772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:57.0970144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:57.1085740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:57.1086272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:57.1089060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:57.1089494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:57.5049483Z dist init r=0, world=2 2022-08-17T12:17:57.5164415Z dist init r=1, world=2 2022-08-17T12:17:57.5359641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:57.5360382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:57.5361191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:57.5361756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:57.5365587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:57.5366380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:57.9328979Z skip: Need at least 2 CUDA devices (1.825s) 2022-08-17T12:17:57.9382734Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10769 2022-08-17T12:17:57.9419327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10770 2022-08-17T12:17:58.9218868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:58.9219365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:58.9220991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:58.9221372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:58.9282013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:17:58.9282379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:17:58.9286047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:17:58.9286482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:17:59.3239356Z dist init r=0, world=2 2022-08-17T12:17:59.3324157Z dist init r=1, world=2 2022-08-17T12:17:59.3449288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:17:59.3449731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:17:59.3450354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:59.3450910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:17:59.3555902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:17:59.3556521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:17:59.7479997Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:17:59.7515419Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10797 2022-08-17T12:17:59.7554476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10798 2022-08-17T12:18:00.7363881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:00.7364369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:00.7366533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:00.7366911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:00.7391971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:00.7392352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:00.7395513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:00.7395926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:01.1454354Z dist init r=0, world=2 2022-08-17T12:18:01.1491701Z dist init r=1, world=2 2022-08-17T12:18:01.1663358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:01.1663916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:01.1664553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:01.1665102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:01.1769622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:01.1770176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:01.5595707Z skip: Need at least 2 CUDA devices (1.814s) 2022-08-17T12:18:01.5650062Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10825 2022-08-17T12:18:01.5688037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10826 2022-08-17T12:18:02.5599712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:02.5600240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:02.5600942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:02.5601391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:02.5601859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:02.5602436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:02.5604172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:02.5604605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:02.9600152Z dist init r=1, world=2 2022-08-17T12:18:02.9607242Z dist init r=0, world=2 2022-08-17T12:18:02.9809916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:02.9810650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:02.9811429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:02.9811979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:02.9816058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:02.9843735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:03.3726442Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:03.3780428Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10853 2022-08-17T12:18:03.3818064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10854 2022-08-17T12:18:04.3580500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:04.3581212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:04.3582094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:04.3582733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:04.3583510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:04.3584139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:04.3585242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:04.3585843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:04.7627870Z dist init r=0, world=2 2022-08-17T12:18:04.7628241Z dist init r=1, world=2 2022-08-17T12:18:04.7837356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:04.7837936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:04.7838614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:04.7839173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:04.7944158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:04.7944610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:05.1855408Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:05.1910515Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10881 2022-08-17T12:18:05.1949550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10882 2022-08-17T12:18:06.1684555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:06.1685446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:06.1687023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:06.1687602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:06.1793533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:06.1794168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:06.1795858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:06.1796545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:06.5747201Z dist init r=0, world=2 2022-08-17T12:18:06.5865315Z dist init r=1, world=2 2022-08-17T12:18:06.6057937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:06.6058799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:06.6059440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:06.6060001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:06.6063357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:06.6063761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:06.9983986Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:07.0039219Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10909 2022-08-17T12:18:07.0078726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10910 2022-08-17T12:18:07.9858536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:07.9859260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:07.9860281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:07.9860929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:07.9934652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:07.9935574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:07.9938603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:07.9939215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:08.3993304Z dist init r=1, world=2 2022-08-17T12:18:08.3997158Z dist init r=0, world=2 2022-08-17T12:18:08.4206796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:08.4207376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:08.4208136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:08.4208702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:08.4312725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:08.4313250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:08.8113482Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:08.8168897Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10937 2022-08-17T12:18:08.8208210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10938 2022-08-17T12:18:09.8019596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:09.8020032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:09.8022323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:09.8022873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:09.8298705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:09.8299134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:09.8301578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:09.8301958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:10.2076622Z dist init r=0, world=2 2022-08-17T12:18:10.2363085Z dist init r=1, world=2 2022-08-17T12:18:10.2487456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:10.2488007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:10.2488712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:10.2489279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:10.2593683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:10.2594231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:10.6242101Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:10.6294976Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10965 2022-08-17T12:18:10.6331805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10966 2022-08-17T12:18:11.6121763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:11.6122516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:11.6123662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:11.6124281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:11.6479843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:11.6482201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:11.6482992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:11.6483368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:12.0133384Z dist init r=1, world=2 2022-08-17T12:18:12.0471701Z dist init r=0, world=2 2022-08-17T12:18:12.0643820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:12.0644574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:12.0645653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:12.0646403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:12.0650010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:12.0650611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:12.4367586Z skip: Need at least 2 CUDA devices (1.812s) 2022-08-17T12:18:12.4422268Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10993 2022-08-17T12:18:12.4459160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10994 2022-08-17T12:18:13.4203950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:13.4204465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:13.4206077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:13.4206592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:13.4207779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:13.4208467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:13.4212246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:13.4212876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:13.8336572Z dist init r=1, world=2 2022-08-17T12:18:13.8336926Z dist init r=0, world=2 2022-08-17T12:18:13.8444742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:13.8446537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:13.8448570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:13.8546612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:13.8551345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:13.8551747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:14.2494240Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:14.2549143Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11021 2022-08-17T12:18:14.2588230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11022 2022-08-17T12:18:15.2498188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:15.2498863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:15.2499861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:15.2500508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:15.2585592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:15.2586014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:15.2588782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:15.2589196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:15.6622277Z dist init r=0, world=2 2022-08-17T12:18:15.6665032Z dist init r=1, world=2 2022-08-17T12:18:15.6830083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:15.6830842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:15.6831901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:15.6832609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:15.6836576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:15.6837149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:16.0626966Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:16.0684690Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11049 2022-08-17T12:18:16.0724019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11050 2022-08-17T12:18:17.0421048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:17.0421520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:17.0422560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:17.0423261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:17.0674282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:17.0674798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:17.0677111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:17.0677689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:17.4428223Z dist init r=1, world=2 2022-08-17T12:18:17.4746971Z dist init r=0, world=2 2022-08-17T12:18:17.4938916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:17.4939615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:17.4940361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:17.4941100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:17.4944733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:17.4945142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:17.8760393Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:17.8815368Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11077 2022-08-17T12:18:17.8852837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11078 2022-08-17T12:18:18.8561110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:18.8561549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:18.8563327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:18.8563968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:18.8584739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:18.8585087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:18.8588301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:18.8588680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:19.2589279Z dist init r=1, world=2 2022-08-17T12:18:19.2589673Z dist init r=0, world=2 2022-08-17T12:18:19.2697430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:19.2698986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:19.2700026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:19.2799524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:19.2804403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:19.2804954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:19.6887894Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:19.6942176Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11105 2022-08-17T12:18:19.6978627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11106 2022-08-17T12:18:20.6776895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:20.6777595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:20.6778429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:20.6779057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:20.6908158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:20.6908836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:20.6910657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:20.6911567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:21.0811655Z dist init r=1, world=2 2022-08-17T12:18:21.0962325Z dist init r=0, world=2 2022-08-17T12:18:21.1172076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:21.1172526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:21.1173186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:21.1173734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:21.1277996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:21.1278542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:21.5014774Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:21.5073086Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11133 2022-08-17T12:18:21.5110936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11134 2022-08-17T12:18:22.4988651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:22.4989202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:22.4991068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:22.4991461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:22.5087631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:22.5088047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:22.5091019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:22.5091560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:22.9035238Z dist init r=0, world=2 2022-08-17T12:18:22.9080679Z dist init r=1, world=2 2022-08-17T12:18:22.9189703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:22.9190219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:22.9190874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:22.9191436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:22.9295231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:22.9295795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:23.3145875Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:23.3199840Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11161 2022-08-17T12:18:23.3237471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11162 2022-08-17T12:18:24.3138495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:24.3138898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:24.3141340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:24.3141977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:24.3173440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:24.3173799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:24.3177137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:24.3177503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:24.7250282Z dist init r=1, world=2 2022-08-17T12:18:24.7256887Z dist init r=0, world=2 2022-08-17T12:18:24.7467680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:24.7468421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:24.7469193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:24.7469747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:24.7574566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:24.7575377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:25.1273846Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:25.1328571Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11189 2022-08-17T12:18:25.1366513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11190 2022-08-17T12:18:26.1448854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:26.1449492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:26.1450751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:26.1451398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:26.1586025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:26.1586654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:26.1588893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:26.1589501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:26.5479810Z dist init r=0, world=2 2022-08-17T12:18:26.5665136Z dist init r=1, world=2 2022-08-17T12:18:26.5790024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:26.5790779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:26.5791434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:26.5791990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:26.5895886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:26.5896296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:26.9402508Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:26.9455654Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11217 2022-08-17T12:18:26.9494421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11218 2022-08-17T12:18:27.9333643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:27.9334089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:27.9335889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:27.9336439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:27.9400824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:27.9401353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:27.9403579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:27.9404150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:28.3434607Z dist init r=0, world=2 2022-08-17T12:18:28.3485793Z dist init r=1, world=2 2022-08-17T12:18:28.3595233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:28.3595850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:28.3596518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:28.3597070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:28.3701772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:28.3702184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:28.7532362Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:28.7588712Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11245 2022-08-17T12:18:28.7629212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11246 2022-08-17T12:18:29.7423344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:29.7423995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:29.7425287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:29.7425937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:29.7745746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:29.7746309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:29.7747353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:29.7747908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:30.1606522Z dist init r=1, world=2 2022-08-17T12:18:30.1798881Z dist init r=0, world=2 2022-08-17T12:18:30.1916183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:30.1916882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:30.1917691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:30.1918239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:30.1922062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:30.1923089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:30.5664549Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:30.5719569Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11273 2022-08-17T12:18:30.5756875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11274 2022-08-17T12:18:31.5530798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:31.5531189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:31.5533474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:31.5534026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:31.5795039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:31.5795734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:31.5797678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:31.5798058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:31.9646694Z dist init r=0, world=2 2022-08-17T12:18:31.9872358Z dist init r=1, world=2 2022-08-17T12:18:31.9980936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:31.9981393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:31.9982046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:31.9982621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:32.0086788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:32.0087390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:32.3791885Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:32.3849832Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11301 2022-08-17T12:18:32.3885744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11302 2022-08-17T12:18:33.3547165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:33.3547596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:33.3549387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:33.3549906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:33.3946522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:33.3946912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:33.3949285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:33.3949702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:33.7581750Z dist init r=1, world=2 2022-08-17T12:18:33.7965405Z dist init r=0, world=2 2022-08-17T12:18:33.8176309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:33.8177074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:33.8177837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:33.8178401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:33.8182284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:33.8182681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:34.1921383Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:34.1976679Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11329 2022-08-17T12:18:34.2013420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11330 2022-08-17T12:18:35.1703617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:35.1704266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:35.1705991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:35.1706354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:35.1716805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:35.1717451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:35.1720102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:35.1720482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:35.5728986Z dist init r=0, world=2 2022-08-17T12:18:35.5752288Z dist init r=1, world=2 2022-08-17T12:18:35.5937230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:35.5938018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:35.5938754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:35.5939624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:35.5942998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:35.5943579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:36.0048959Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:36.0103313Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11357 2022-08-17T12:18:36.0142532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11358 2022-08-17T12:18:37.0134866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:37.0136293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:37.0137160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:37.0137641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:37.0211683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:37.0212354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:37.0214656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:37.0215220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:37.4216012Z dist init r=0, world=2 2022-08-17T12:18:37.4255396Z dist init r=1, world=2 2022-08-17T12:18:37.4364306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:37.4364727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:37.4365378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:37.4366100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:37.4471122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:37.4471749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:37.8177810Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:37.8233058Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11385 2022-08-17T12:18:37.8271583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11386 2022-08-17T12:18:38.8203920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:38.8204564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:38.8205397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:38.8206265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:38.8251208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:38.8251877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:38.8255382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:38.8255996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:39.2289060Z dist init r=1, world=2 2022-08-17T12:18:39.2289393Z dist init r=0, world=2 2022-08-17T12:18:39.2397438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:39.2399525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:39.2400576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:39.2500104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:39.2504129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:39.2504838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:39.6307236Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:39.6361561Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11413 2022-08-17T12:18:39.6400349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11414 2022-08-17T12:18:40.6234662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:40.6235365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:40.6236728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:40.6237275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:40.6321431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:40.6321795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:40.6324260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:40.6324697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:41.0288017Z dist init r=1, world=2 2022-08-17T12:18:41.0370157Z dist init r=0, world=2 2022-08-17T12:18:41.0497745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:41.0498423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:41.0499244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:41.0499831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:41.0503297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:41.0503704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:41.4439453Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:41.4493926Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11441 2022-08-17T12:18:41.4532136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11442 2022-08-17T12:18:42.4322320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:42.4323006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:42.4324621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:42.4325206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:42.4374787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:42.4375482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:42.4377610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:42.4378282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:42.8359122Z dist init r=1, world=2 2022-08-17T12:18:42.8412360Z dist init r=0, world=2 2022-08-17T12:18:42.8567930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:42.8568403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:42.8569137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:42.8569680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:42.8674207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:42.8674666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:43.2570629Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:43.2626981Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11469 2022-08-17T12:18:43.2666725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11470 2022-08-17T12:18:44.2459468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:44.2460122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:44.2460951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:44.2461560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:44.2762868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:44.2763373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:44.2765175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:44.2766139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:44.6516416Z dist init r=1, world=2 2022-08-17T12:18:44.6815336Z dist init r=0, world=2 2022-08-17T12:18:44.7025139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:44.7025744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:44.7026478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:44.7027041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:44.7030174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:44.7030837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:45.0706197Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:45.0759411Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11497 2022-08-17T12:18:45.0796213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11498 2022-08-17T12:18:46.0568005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:46.0568674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:46.0570049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:46.0570654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:46.0573796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:46.0574434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:46.0577508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:46.0577905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:46.4585392Z dist init r=0, world=2 2022-08-17T12:18:46.4585606Z dist init r=1, world=2 2022-08-17T12:18:46.4694515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:46.4695264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:46.4696520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:46.4697078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:46.4801100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:46.4801633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:46.8834060Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:46.8888136Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11525 2022-08-17T12:18:46.8925137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11526 2022-08-17T12:18:47.8529318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:47.8529711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:47.8531257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:47.8531802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:47.8688418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:47.8688790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:47.8691491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:47.8692157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:48.2535204Z dist init r=1, world=2 2022-08-17T12:18:48.2703690Z dist init r=0, world=2 2022-08-17T12:18:48.2913148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:48.2913896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:48.2914723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:48.2915288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:48.2919092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:48.2919744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:48.6964539Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:48.7020353Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11553 2022-08-17T12:18:48.7058105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11554 2022-08-17T12:18:49.6827392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:49.6827967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:49.6829227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:49.6829665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:49.7304179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:49.7304831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:49.7306328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:49.7307232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:50.0840155Z dist init r=1, world=2 2022-08-17T12:18:50.1317618Z dist init r=0, world=2 2022-08-17T12:18:50.1527837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:50.1528335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:50.1529059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:50.1529614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:50.1533229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:50.1533624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:50.5096375Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:50.5142270Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11581 2022-08-17T12:18:50.5173247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11582 2022-08-17T12:18:51.5040106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:51.5040559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:51.5041879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:51.5042466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:51.5353817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:51.5354208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:51.5356837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:51.5357227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:51.9105740Z dist init r=1, world=2 2022-08-17T12:18:51.9394769Z dist init r=0, world=2 2022-08-17T12:18:51.9605055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:51.9605713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:51.9606836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:51.9607501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:51.9610398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:51.9611014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:52.3208252Z skip: Need at least 2 CUDA devices (1.811s) 2022-08-17T12:18:52.3262457Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11609 2022-08-17T12:18:52.3299736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11610 2022-08-17T12:18:53.3214295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:53.3214952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:53.3215777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:53.3216637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:53.3217377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:53.3218013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:53.3219178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:53.3219771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:53.7310113Z dist init r=0, world=2 2022-08-17T12:18:53.7359663Z dist init r=1, world=2 2022-08-17T12:18:53.7469074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:53.7469544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:53.7470395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:53.7471230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:53.7475568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:53.7476189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:54.1335649Z skip: Need at least 2 CUDA devices (1.812s) 2022-08-17T12:18:54.1389794Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11637 2022-08-17T12:18:54.1428347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11638 2022-08-17T12:18:55.1353607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:55.1354075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:55.1355676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:55.1356213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:55.1359370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:55.1360007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:55.1363201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:55.1363649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:55.5383458Z dist init r=0, world=2 2022-08-17T12:18:55.5412017Z dist init r=1, world=2 2022-08-17T12:18:55.5521104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:55.5521660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:55.5522335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:55.5522977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:55.5527071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:55.5527483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:55.9462767Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:18:55.9478441Z test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded) 2022-08-17T12:18:55.9479364Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.002s) 2022-08-17T12:18:55.9543387Z test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11665 2022-08-17T12:18:55.9580251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11666 2022-08-17T12:18:56.9406316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:56.9407005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:56.9408321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:56.9408928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:56.9494264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:56.9494937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:56.9496874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:56.9497543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:57.3470713Z dist init r=0, world=2 2022-08-17T12:18:57.3556067Z dist init r=1, world=2 2022-08-17T12:18:57.3679328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:57.3679976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:57.3680763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:57.3681637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:57.3685341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:57.3686275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:57.7618426Z skip: Need at least 2 CUDA devices (1.814s) 2022-08-17T12:18:57.7688661Z test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11693 2022-08-17T12:18:57.7727836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11694 2022-08-17T12:18:58.7502283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:58.7502707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:58.7504182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:58.7504728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:58.7777958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:18:58.7778356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:18:58.7780588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:18:58.7780995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:18:59.1628712Z dist init r=1, world=2 2022-08-17T12:18:59.1923105Z dist init r=0, world=2 2022-08-17T12:18:59.2039203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:18:59.2039821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:18:59.2040502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:59.2041383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:18:59.2044926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:18:59.2045464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:18:59.5763198Z skip: Need at least 2 CUDA devices (1.814s) 2022-08-17T12:18:59.5816469Z test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11721 2022-08-17T12:18:59.5854943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11722 2022-08-17T12:19:00.5695547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:00.5695958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:00.5698216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:00.5698614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:00.5991143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:00.5991863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:00.5994356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:00.5994924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:00.9819953Z dist init r=0, world=2 2022-08-17T12:19:00.9978688Z dist init r=1, world=2 2022-08-17T12:19:01.0087704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:01.0088479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:01.0089176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:01.0089734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:01.0193840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:01.0194440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:01.3888844Z skip: Need at least 2 CUDA devices (1.812s) 2022-08-17T12:19:01.3945421Z test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11749 2022-08-17T12:19:01.3983502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11750 2022-08-17T12:19:02.3879094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:02.3880524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:02.3881405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:02.3882126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:02.3938194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:02.3938890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:02.3941830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:02.3942419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:02.7934955Z dist init r=0, world=2 2022-08-17T12:19:02.8011767Z dist init r=1, world=2 2022-08-17T12:19:02.8144729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:02.8145321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:02.8145972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:02.8146534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:02.8250682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:02.8251218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:03.2022100Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:19:03.2076168Z test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11777 2022-08-17T12:19:03.2114112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11778 2022-08-17T12:19:04.1889302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:04.1889880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:04.1892000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:04.1892591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:04.2079994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:04.2080705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:04.2082071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:04.2082761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:04.5940206Z dist init r=1, world=2 2022-08-17T12:19:04.6133118Z dist init r=0, world=2 2022-08-17T12:19:04.6249472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:04.6250021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:04.6250678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:04.6251236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:04.6257160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:04.6257736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:05.0150031Z skip: Need at least 2 CUDA devices (1.813s) 2022-08-17T12:19:05.0201527Z test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11805 2022-08-17T12:19:05.0239247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11806 2022-08-17T12:19:06.0052020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:06.0052656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:06.0053818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:06.0054445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:06.0146595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:06.0147237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:06.0149687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:06.0150362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:06.4109719Z dist init r=1, world=2 2022-08-17T12:19:06.4239485Z dist init r=0, world=2 2022-08-17T12:19:06.4450377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:06.4450836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:06.4451494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:06.4452040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:19:06.4556992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:06.4557606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:06.8275458Z skip: Need at least 2 CUDA devices (1.812s) 2022-08-17T12:19:06.8334434Z test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11833 2022-08-17T12:19:07.8039505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:07.8039894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:07.8041788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:07.8042256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:08.2040815Z dist init r=0, world=1 2022-08-17T12:19:08.2049133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:08.2050130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:19:08.2053080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:08.5367025Z skip: Need at least 1 CUDA device (1.709s) 2022-08-17T12:19:08.5422006Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11847 2022-08-17T12:19:09.5026914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:09.5027371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:09.5028567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:09.5029043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:09.8937577Z dist init r=0, world=1 2022-08-17T12:19:09.8945808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:09.8946698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:19:09.8950440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:10.2454525Z skip: Need at least 1 CUDA device (1.709s) 2022-08-17T12:19:10.2454774Z 2022-08-17T12:19:10.2455174Z ---------------------------------------------------------------------- 2022-08-17T12:19:10.2455567Z Ran 42 tests in 74.138s 2022-08-17T12:19:10.2455761Z 2022-08-17T12:19:10.2455827Z OK (skipped=42) 2022-08-17T12:19:10.2458866Z 2022-08-17T12:19:10.2459092Z Generating XML reports... 2022-08-17T12:19:10.2540219Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220817121756.xml 2022-08-17T12:19:10.2543648Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220817121756.xml 2022-08-17T12:19:10.5855821Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-08-17 12:19:10.585159] 2022-08-17T12:19:10.5856450Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:19:10.585248] 2022-08-17T12:19:11.7216314Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward 2022-08-17T12:19:11.7228075Z 2022-08-17T12:19:11.7228182Z Running tests... 2022-08-17T12:19:11.7229106Z ---------------------------------------------------------------------- 2022-08-17T12:19:11.9837301Z test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11872 2022-08-17T12:19:11.9865419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11873 2022-08-17T12:19:11.9893021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11874 2022-08-17T12:19:11.9923652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11875 2022-08-17T12:19:13.0675177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:13.0675821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:13.0676925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:13.0678255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:13.0682108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:13.0682788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:13.0686258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:13.0686922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:13.0837714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:13.0838302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:13.0839361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:13.0840001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:13.0948045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:13.0948657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:13.0951477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:13.0952035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:13.2273341Z dist init r=3, world=4 2022-08-17T12:19:13.2462031Z dist init r=0, world=4 2022-08-17T12:19:13.2462401Z dist init r=2, world=4 2022-08-17T12:19:13.2540447Z dist init r=1, world=4 2022-08-17T12:19:13.2682527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:13.2783495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:13.2872920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:13.2873648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:13.2874846Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:13.2875400Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:13.2886635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:13.2887229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:13.2982958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:13.2983558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:13.2984119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:13.2984702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:13.5968272Z skip: Need at least 2 CUDA devices (1.874s) 2022-08-17T12:19:13.5968519Z 2022-08-17T12:19:13.5968837Z ---------------------------------------------------------------------- 2022-08-17T12:19:13.5969185Z Ran 1 test in 1.874s 2022-08-17T12:19:13.5969305Z 2022-08-17T12:19:13.5969601Z OK (skipped=1) 2022-08-17T12:19:13.5969762Z 2022-08-17T12:19:13.5971139Z Generating XML reports... 2022-08-17T12:19:13.6002475Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220817121911.xml 2022-08-17T12:19:13.9106256Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-08-17 12:19:13.910202] 2022-08-17T12:19:13.9106892Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:19:13.910293] 2022-08-17T12:19:15.0413106Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping 2022-08-17T12:19:15.0425271Z 2022-08-17T12:19:15.0425371Z Running tests... 2022-08-17T12:19:15.0425975Z ---------------------------------------------------------------------- 2022-08-17T12:19:15.0436170Z test_multiple_wrapping (__main__.TestMultipleWrapping) 2022-08-17T12:19:15.3114783Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11939 2022-08-17T12:19:15.3143177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11940 2022-08-17T12:19:15.3172059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11941 2022-08-17T12:19:15.3204006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11942 2022-08-17T12:19:16.3568769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:16.3569469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:16.3570535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:16.3571164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:16.3693563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:16.3694231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:16.3696372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:16.3697254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:16.3818517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:16.3819179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:16.3820230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:16.3820851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:16.3860901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:16.3861537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:16.3865045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:16.3865704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:16.5147611Z dist init r=1, world=4 2022-08-17T12:19:16.5273683Z dist init r=0, world=4 2022-08-17T12:19:16.5385020Z dist init r=2, world=4 2022-08-17T12:19:16.5437248Z dist init r=3, world=4 2022-08-17T12:19:16.5545683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:16.5646842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:16.5748820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:16.5749627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:16.5750750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:16.5751511Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:16.5752179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:16.5752878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:16.5760314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:16.5761127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:16.5761734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:16.5765327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:16.9247709Z skip: Need at least 2 CUDA devices (1.882s) 2022-08-17T12:19:16.9248006Z 2022-08-17T12:19:16.9248525Z ---------------------------------------------------------------------- 2022-08-17T12:19:16.9248895Z Ran 1 test in 1.882s 2022-08-17T12:19:16.9249017Z 2022-08-17T12:19:16.9249099Z OK (skipped=1) 2022-08-17T12:19:16.9249213Z 2022-08-17T12:19:16.9249292Z Generating XML reports... 2022-08-17T12:19:16.9282801Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220817121915.xml 2022-08-17T12:19:17.2475470Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-08-17 12:19:17.247119] 2022-08-17T12:19:17.2476400Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:19:17.247212] 2022-08-17T12:19:18.3819831Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state 2022-08-17T12:19:18.3834033Z 2022-08-17T12:19:18.3834397Z Running tests... 2022-08-17T12:19:18.3834854Z ---------------------------------------------------------------------- 2022-08-17T12:19:18.3846288Z test_full_optim_state_dict_keys (__main__.TestFSDPOptimState) 2022-08-17T12:19:18.6442945Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12006 2022-08-17T12:19:18.6470813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12007 2022-08-17T12:19:18.6498780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12008 2022-08-17T12:19:18.6529291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12009 2022-08-17T12:19:19.6740282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:19.6740962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:19.6742477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:19.6743100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:19.7110269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:19.7110950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:19.7112464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:19.7113129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:19.7212382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:19.7213241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:19.7214859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:19.7215500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:19.7368981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:19.7369642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:19.7371758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:19.7372417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:19.8321055Z dist init r=3, world=4 2022-08-17T12:19:19.8689338Z dist init r=2, world=4 2022-08-17T12:19:19.8817754Z dist init r=0, world=4 2022-08-17T12:19:19.8951303Z dist init r=1, world=4 2022-08-17T12:19:19.9200957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:19.9301899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:19.9302855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:19.9303672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:19.9304704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:19.9305534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:19.9306221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:19.9306953Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:19.9410034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:19.9410619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:19.9411194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:19.9411773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:20.2572652Z skip: Need at least 2 CUDA devices (1.874s) 2022-08-17T12:19:20.2581654Z test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState) 2022-08-17T12:19:20.2629308Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12062 2022-08-17T12:19:20.2666175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12063 2022-08-17T12:19:20.2695433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12064 2022-08-17T12:19:20.2725054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12065 2022-08-17T12:19:21.3012341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:21.3012919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:21.3015482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:21.3016172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:21.3022856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:21.3023514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:21.3027422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:21.3028043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:21.3546792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:21.3547412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:21.3550001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:21.3550606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:21.3808019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:21.3808527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:21.3810403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:21.3811549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:21.4616434Z dist init r=1, world=4 2022-08-17T12:19:21.4624012Z dist init r=2, world=4 2022-08-17T12:19:21.5189084Z dist init r=0, world=4 2022-08-17T12:19:21.5538507Z dist init r=3, world=4 2022-08-17T12:19:21.5742297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:21.5742817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:21.5844459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:21.5845101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:21.5845795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:21.5846678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:21.5847292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:21.5847904Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:21.5951468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:21.5952069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:21.5952624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:21.5952988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:21.8764677Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:21.8780078Z test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:21.8828995Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12118 2022-08-17T12:19:21.8866731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12119 2022-08-17T12:19:21.8897665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12120 2022-08-17T12:19:21.8926985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12121 2022-08-17T12:19:22.8881397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:22.8882081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:22.8883719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:22.8884379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:22.8950541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:22.8951217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:22.8953975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:22.8954650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:22.9010363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:22.9011061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:22.9014579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:22.9015254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:22.9510086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:22.9512374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:22.9513302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:22.9513923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:23.0499155Z dist init r=2, world=4 2022-08-17T12:19:23.0546176Z dist init r=3, world=4 2022-08-17T12:19:23.0609609Z dist init r=0, world=4 2022-08-17T12:19:23.1114667Z dist init r=1, world=4 2022-08-17T12:19:23.1312099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:23.1514411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:23.1616620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:23.1618312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:23.1619339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:23.1619899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:23.1620702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:23.1621748Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:23.1625463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:23.1626052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:23.1626655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:23.1627199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:23.4967844Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:23.4982449Z test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:23.5033221Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12174 2022-08-17T12:19:23.5071118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12175 2022-08-17T12:19:23.5099681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12176 2022-08-17T12:19:23.5128843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12177 2022-08-17T12:19:24.5009065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:24.5009851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:24.5011285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:24.5011896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:24.5023455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:24.5024101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:24.5026826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:24.5027477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:24.5040709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:24.5041272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:24.5045192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:24.5046101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:24.5686067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:24.5686678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:24.5688187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:24.5688830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:24.6619771Z dist init r=3, world=4 2022-08-17T12:19:24.6652722Z dist init r=1, world=4 2022-08-17T12:19:24.6825641Z dist init r=2, world=4 2022-08-17T12:19:24.7307918Z dist init r=0, world=4 2022-08-17T12:19:24.7539582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:24.7642576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:24.7643055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:24.7643452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:24.7644279Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:24.7644841Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:24.7645399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:24.7646091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:24.7749325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:24.7750154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:24.7750841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:24.7751512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:25.1170124Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:25.1183783Z test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:25.1232075Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12230 2022-08-17T12:19:25.1268640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12231 2022-08-17T12:19:25.1297750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12232 2022-08-17T12:19:25.1327509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12233 2022-08-17T12:19:26.1190790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:26.1191504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:26.1192635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:26.1193275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:26.1529954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:26.1530527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:26.1532353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:26.1532836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:26.1649571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:26.1650232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:26.1652922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:26.1653602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:26.1849997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:26.1850545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:26.1852488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:26.1853008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:26.2766910Z dist init r=3, world=4 2022-08-17T12:19:26.3135386Z dist init r=2, world=4 2022-08-17T12:19:26.3246310Z dist init r=0, world=4 2022-08-17T12:19:26.3418960Z dist init r=1, world=4 2022-08-17T12:19:26.3646711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:26.3731074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:26.3731739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:26.3732157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:26.3732816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:26.3733356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:26.3733906Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:26.3748676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:26.3839558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:26.3840326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:26.3841259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:26.3841843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:26.7367347Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:26.7383300Z test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:26.7424674Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12286 2022-08-17T12:19:26.7457243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12287 2022-08-17T12:19:26.7487078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12288 2022-08-17T12:19:26.7526112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12289 2022-08-17T12:19:27.8194025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:27.8194684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:27.8195525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:27.8196154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:27.8200647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:27.8201269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:27.8204546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:27.8205209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:27.8262736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:27.8263393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:27.8266388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:27.8267054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:27.8511550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:27.8512217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:27.8513838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:27.8514787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:27.9811455Z dist init r=3, world=4 2022-08-17T12:19:27.9811832Z dist init r=0, world=4 2022-08-17T12:19:27.9861271Z dist init r=2, world=4 2022-08-17T12:19:28.0107418Z dist init r=1, world=4 2022-08-17T12:19:28.0373196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:28.0474636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:28.0475415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:28.0476292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:28.0476814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:28.0477553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:28.0478194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:28.0479253Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:28.0484055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:28.0484743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:28.0485418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:28.0486269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:28.3565693Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:28.3580647Z test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:28.3628074Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12342 2022-08-17T12:19:28.3665031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12343 2022-08-17T12:19:28.3693815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12344 2022-08-17T12:19:28.3723100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12345 2022-08-17T12:19:29.3932442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:29.3933072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:29.3934259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:29.3934898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:29.4026935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:29.4027623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:29.4030236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:29.4030892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:29.4063525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:29.4064180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:29.4066177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:29.4067076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:29.4214104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:29.4214665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:29.4216461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:29.4216993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:29.5532691Z dist init r=1, world=4 2022-08-17T12:19:29.5641215Z dist init r=3, world=4 2022-08-17T12:19:29.5647049Z dist init r=2, world=4 2022-08-17T12:19:29.5809853Z dist init r=0, world=4 2022-08-17T12:19:29.6059429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:29.6161796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:29.6162447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:29.6162946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:29.6163782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:29.6164341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:29.6164890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:29.6165417Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:29.6172258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:29.6172742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:29.6173311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:29.6174410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:29.9763674Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:29.9779239Z test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:29.9827063Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12398 2022-08-17T12:19:29.9865705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12399 2022-08-17T12:19:29.9897246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12400 2022-08-17T12:19:29.9928028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12401 2022-08-17T12:19:31.0303770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:31.0304471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:31.0305838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:31.0306443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:31.0423894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:31.0424580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:31.0425446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:31.0426106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:31.0429075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:31.0429691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:31.0433088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:31.0433773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:31.0440293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:31.0440941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:31.0444645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:31.0446716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:31.1897747Z dist init r=3, world=4 2022-08-17T12:19:31.2031373Z dist init r=1, world=4 2022-08-17T12:19:31.2031788Z dist init r=0, world=4 2022-08-17T12:19:31.2032013Z dist init r=2, world=4 2022-08-17T12:19:31.2207244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:31.2308829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:31.2410824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:31.2412205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:31.2412888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:31.2413534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:31.2414334Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:31.2415201Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:31.2421995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:31.2422598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:31.2423190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:31.2423731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:31.5969125Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:31.5983142Z test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:31.6032331Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12454 2022-08-17T12:19:31.6071871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12455 2022-08-17T12:19:31.6102360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12456 2022-08-17T12:19:31.6134170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12457 2022-08-17T12:19:32.6431063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:32.6432965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:32.6433858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:32.6434455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:32.6528794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:32.6529695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:32.6531274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:32.6531939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:32.6635081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:32.6635695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:32.6638194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:32.6638851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:32.6695552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:32.6696119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:32.6699240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:32.6700183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:32.8039631Z dist init r=1, world=4 2022-08-17T12:19:32.8124630Z dist init r=3, world=4 2022-08-17T12:19:32.8215219Z dist init r=0, world=4 2022-08-17T12:19:32.8280978Z dist init r=2, world=4 2022-08-17T12:19:32.8434144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:32.8626720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:32.8728574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:32.8729050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:32.8729708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:32.8730276Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:32.8730833Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:32.8738501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:32.8837553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:32.8838375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:32.8838975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:32.8839538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:33.2174750Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:33.2189898Z test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:33.2237555Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12510 2022-08-17T12:19:33.2274331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12511 2022-08-17T12:19:33.2303119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12512 2022-08-17T12:19:33.2331779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12513 2022-08-17T12:19:34.2632615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:34.2633610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:34.2635103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:34.2635811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:34.2764145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:34.2764798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:34.2766586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:34.2767198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:34.2868851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:34.2869544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:34.2871897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:34.2872726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:34.3042763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:34.3043536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:34.3044354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:34.3044992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:34.4234987Z dist init r=2, world=4 2022-08-17T12:19:34.4379691Z dist init r=0, world=4 2022-08-17T12:19:34.4465222Z dist init r=3, world=4 2022-08-17T12:19:34.4616868Z dist init r=1, world=4 2022-08-17T12:19:34.4877066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:34.4877500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:34.4977967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:34.4978703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:34.4979373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:34.4981390Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:34.4981948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:34.4982507Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:34.5086960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:34.5087418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:34.5087982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:34.5088545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:34.8373221Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:34.8384870Z test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:34.8432432Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12566 2022-08-17T12:19:34.8470560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12567 2022-08-17T12:19:34.8500103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12568 2022-08-17T12:19:34.8529528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12569 2022-08-17T12:19:35.8944358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:35.8946116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:35.8947018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:35.8948500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:35.8949400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:35.8950127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:35.8950969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:35.8951635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:35.8963450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:35.8964402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:35.8967180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:35.8967821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:35.9090657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:35.9091473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:35.9093199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:35.9093861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:36.0573552Z dist init r=3, world=4 2022-08-17T12:19:36.0577550Z dist init r=0, world=4 2022-08-17T12:19:36.0597653Z dist init r=2, world=4 2022-08-17T12:19:36.0693287Z dist init r=1, world=4 2022-08-17T12:19:36.1006604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:36.1007038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:36.1107242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:36.1107837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:36.1108655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:36.1109530Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:36.1110271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:36.1110835Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:36.1216041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:36.1216677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:36.1217204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:36.1217759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:36.4571430Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:36.4583907Z test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:36.4633194Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12622 2022-08-17T12:19:36.4671549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12623 2022-08-17T12:19:36.4701711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12624 2022-08-17T12:19:36.4731238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12625 2022-08-17T12:19:37.5711190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:37.5711868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:37.5713459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:37.5714082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:37.5760957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:37.5761619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:37.5764647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:37.5765376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:37.6244031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:37.6244789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:37.6246596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:37.6247341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:37.6263625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:37.6264279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:37.6266331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:37.6266962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:37.7308205Z dist init r=0, world=4 2022-08-17T12:19:37.7348235Z dist init r=1, world=4 2022-08-17T12:19:37.7826332Z dist init r=2, world=4 2022-08-17T12:19:37.7866278Z dist init r=3, world=4 2022-08-17T12:19:37.7976141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:37.8076498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:37.8163474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:37.8164163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:37.8164830Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:37.8165395Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:37.8178732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:37.8179469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:37.8270735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:37.8271317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:37.8272110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:37.8272714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:38.1773723Z skip: Need at least 2 CUDA devices (1.720s) 2022-08-17T12:19:38.1786135Z test_rekey_optim_state_dict_to_names_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:38.1834544Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12678 2022-08-17T12:19:38.1873021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12679 2022-08-17T12:19:38.1902493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12680 2022-08-17T12:19:38.1933545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12681 2022-08-17T12:19:39.2180496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:39.2181157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:39.2183950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:39.2186366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:39.2582767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:39.2583317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:39.2585094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:39.2585665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:39.2589528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:39.2590293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:39.2593279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:39.2593943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:39.2648864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:39.2649489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:39.2652626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:39.2653306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:39.3770916Z dist init r=0, world=4 2022-08-17T12:19:39.4223263Z dist init r=2, world=4 2022-08-17T12:19:39.4269712Z dist init r=1, world=4 2022-08-17T12:19:39.4363631Z dist init r=3, world=4 2022-08-17T12:19:39.4585212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:39.4687423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:39.4688136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:39.4688789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:39.4689532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:39.4690093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:39.4690630Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:39.4691379Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:39.4794246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:39.4794862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:39.4795224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:39.4795676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:39.7974437Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:39.7980388Z test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-08-17T12:19:39.8029337Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12734 2022-08-17T12:19:39.8068332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12735 2022-08-17T12:19:39.8099611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12736 2022-08-17T12:19:39.8129616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12737 2022-08-17T12:19:40.8108335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:40.8109063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:40.8109879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:40.8110578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:40.8111342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:40.8111962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:40.8112772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:40.8113379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:40.8465098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:40.8465815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:40.8466948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:40.8467587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:40.8694578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:40.8695326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:40.8696785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:40.8697520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:40.9735280Z dist init r=2, world=4 2022-08-17T12:19:40.9735658Z dist init r=0, world=4 2022-08-17T12:19:41.0082825Z dist init r=3, world=4 2022-08-17T12:19:41.0274400Z dist init r=1, world=4 2022-08-17T12:19:41.0448176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:41.0548036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:41.0596041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:41.0596464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:41.0597102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:41.0597836Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:41.0650546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:41.0651465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:41.0757729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:41.0758466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:41.0760224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:41.0760777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:41.4170704Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:41.4178643Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:41.4227577Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12790 2022-08-17T12:19:41.4265368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12791 2022-08-17T12:19:41.4294394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12792 2022-08-17T12:19:41.4323601Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12793 2022-08-17T12:19:42.4616497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:42.4617364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:42.4618848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:42.4619461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:42.4788945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:42.4789582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:42.4791261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:42.4791926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:42.4920800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:42.4921480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:42.4923038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:42.4923698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:42.5114155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:42.5114793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:42.5116077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:42.5116713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:42.6382106Z dist init r=1, world=4 2022-08-17T12:19:42.6410569Z dist init r=2, world=4 2022-08-17T12:19:42.6534249Z dist init r=0, world=4 2022-08-17T12:19:42.6722172Z dist init r=3, world=4 2022-08-17T12:19:42.6894914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:42.6895493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:42.6996803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:42.6997463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:42.6998465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:42.6999520Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:42.7000079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:42.7000629Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:42.7007114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:42.7007727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:42.7008298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:42.7009309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:43.0363886Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:43.0371295Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:43.0420728Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12846 2022-08-17T12:19:43.0459369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12847 2022-08-17T12:19:43.0491270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12848 2022-08-17T12:19:43.0522219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12849 2022-08-17T12:19:44.0640889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:44.0641571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:44.0643328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:44.0643987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:44.0874161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:44.0874833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:44.0875628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:44.0876276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:44.0893697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:44.0894365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:44.0896908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:44.0897605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:44.1097902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:44.1098516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:44.1100142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:44.1100769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:44.2249427Z dist init r=3, world=4 2022-08-17T12:19:44.2491293Z dist init r=2, world=4 2022-08-17T12:19:44.2507760Z dist init r=0, world=4 2022-08-17T12:19:44.2700163Z dist init r=1, world=4 2022-08-17T12:19:44.2902076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:44.2960995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:44.3063501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:44.3064240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:44.3064679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:44.3065196Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:44.3065765Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:44.3105575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:44.3172787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:44.3173498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:44.3174169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:44.3174842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:44.6562115Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:44.6569530Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:44.6611871Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12902 2022-08-17T12:19:44.6643817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12903 2022-08-17T12:19:44.6674207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12904 2022-08-17T12:19:44.6712251Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12905 2022-08-17T12:19:45.7133350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:45.7134033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:45.7134888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:45.7135511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:45.7229429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:45.7230194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:45.7232210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:45.7232847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:45.7373210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:45.7373914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:45.7375414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:45.7376088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:45.7391911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:45.7392452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:45.7395693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:45.7396259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:45.8758739Z dist init r=3, world=4 2022-08-17T12:19:45.8845463Z dist init r=2, world=4 2022-08-17T12:19:45.8979591Z dist init r=0, world=4 2022-08-17T12:19:45.8983615Z dist init r=1, world=4 2022-08-17T12:19:45.9170469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:45.9271312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:45.9373289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:45.9374015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:45.9374732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:45.9375462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:45.9376003Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:45.9376544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:45.9480732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:45.9481598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:45.9482280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:45.9482959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:46.2752747Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:46.2760179Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:46.2808759Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12958 2022-08-17T12:19:46.2846222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12959 2022-08-17T12:19:46.2876396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12960 2022-08-17T12:19:46.2907164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12961 2022-08-17T12:19:47.2725659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:47.2726567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:47.2727880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:47.2728479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:47.2772454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:47.2773111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:47.2775801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:47.2776499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:47.2966320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:47.2967324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:47.2968873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:47.2969548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:47.2973706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:47.2974203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:47.2977849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:47.2978379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:47.4333689Z dist init r=2, world=4 2022-08-17T12:19:47.4379435Z dist init r=3, world=4 2022-08-17T12:19:47.4548944Z dist init r=1, world=4 2022-08-17T12:19:47.4574971Z dist init r=0, world=4 2022-08-17T12:19:47.4961288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:47.4961714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:47.5062630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:47.5063784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:47.5064502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:47.5065283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:47.5065831Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:47.5066388Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:47.5169622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:47.5171776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:47.5172369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:47.5172929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:47.7946635Z skip: Need at least 2 CUDA devices (1.519s) 2022-08-17T12:19:47.7953603Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:47.8001387Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13014 2022-08-17T12:19:47.8038528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13015 2022-08-17T12:19:47.8068409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13016 2022-08-17T12:19:47.8108036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13017 2022-08-17T12:19:48.7910197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:48.7910890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:48.7911732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:48.7912352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:48.8257981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:48.8258832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:48.8259748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:48.8260416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:48.8313645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:48.8314251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:48.8316533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:48.8317186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:48.8913298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:48.8913849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:48.8915894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:48.8916474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:48.9528581Z dist init r=2, world=4 2022-08-17T12:19:48.9887994Z dist init r=3, world=4 2022-08-17T12:19:48.9925449Z dist init r=0, world=4 2022-08-17T12:19:49.0487366Z dist init r=1, world=4 2022-08-17T12:19:49.0702907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:49.0905360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:49.1007186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:49.1007692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:49.1008538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:49.1009087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:49.1009641Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:49.1010190Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:49.1115619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:49.1116207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:49.1116769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:49.1117858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:49.4148465Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:19:49.4156514Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:49.4204535Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13070 2022-08-17T12:19:49.4241089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13071 2022-08-17T12:19:49.4270686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13072 2022-08-17T12:19:49.4300771Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13073 2022-08-17T12:19:50.4261994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:50.4263335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:50.4264443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:50.4265046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:50.4453902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:50.4454505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:50.4456056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:50.4456701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:50.4817266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:50.4817876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:50.4819456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:50.4820034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:50.4946827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:50.4947509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:50.4948879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:50.4949536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:50.5863861Z dist init r=0, world=4 2022-08-17T12:19:50.6055213Z dist init r=2, world=4 2022-08-17T12:19:50.6406882Z dist init r=1, world=4 2022-08-17T12:19:50.6531815Z dist init r=3, world=4 2022-08-17T12:19:50.6641088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:50.6718061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:50.6718555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:50.6719349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:50.6720025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:50.6720597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:50.6721151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:50.6742970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:50.6826101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:50.6826646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:50.6827244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:50.6827787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:51.0341900Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:51.0349186Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:51.0401238Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13126 2022-08-17T12:19:51.0439585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13127 2022-08-17T12:19:51.0468914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13128 2022-08-17T12:19:51.0498679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13129 2022-08-17T12:19:52.0382108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:52.0382854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:52.0383679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:52.0384344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:52.0475979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:52.0476602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:52.0479246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:52.0479896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:52.0497101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:52.0498023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:52.0501611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:52.0909199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:52.0910214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:52.0910838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:52.0911858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:52.0912496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:52.1977887Z dist init r=3, world=4 2022-08-17T12:19:52.2096593Z dist init r=1, world=4 2022-08-17T12:19:52.2101294Z dist init r=0, world=4 2022-08-17T12:19:52.2484658Z dist init r=2, world=4 2022-08-17T12:19:52.2712589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:52.2713363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:52.2713819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:52.2714300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:52.2715004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:52.2715613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:52.2716178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:52.2716802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:52.2820481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:52.2821062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:52.2821438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:52.2821804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:52.6538562Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:52.6546151Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:52.6594542Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13182 2022-08-17T12:19:52.6632743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13183 2022-08-17T12:19:52.6662928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13184 2022-08-17T12:19:52.6692961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13185 2022-08-17T12:19:53.6726102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:53.6726766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:53.6728301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:53.6728924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:53.6747366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:53.6747986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:53.6751160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:53.6751855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:53.6843186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:53.6843859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:53.6846489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:53.6847145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:53.7193095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:53.7193716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:53.7195318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:53.7195888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:53.8310956Z dist init r=1, world=4 2022-08-17T12:19:53.8341141Z dist init r=3, world=4 2022-08-17T12:19:53.8436542Z dist init r=2, world=4 2022-08-17T12:19:53.8751294Z dist init r=0, world=4 2022-08-17T12:19:53.9162528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:53.9162954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:53.9263997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:53.9265123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:53.9265782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:53.9266482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:53.9267303Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:53.9268168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:53.9271552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:53.9272175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:53.9272962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:53.9273560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:54.2733406Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:54.2739122Z test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-08-17T12:19:54.2787439Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13238 2022-08-17T12:19:54.2824612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13239 2022-08-17T12:19:54.2854066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13240 2022-08-17T12:19:54.2882926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13241 2022-08-17T12:19:55.3012762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:55.3013419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:55.3015119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:55.3015977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:55.3176230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:55.3176842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:55.3178824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:55.3179426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:55.3506124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:55.3506705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:55.3508165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:55.3508699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:55.3569816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:55.3570385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:55.3573809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:55.3574351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:55.4620418Z dist init r=1, world=4 2022-08-17T12:19:55.4777731Z dist init r=3, world=4 2022-08-17T12:19:55.5104140Z dist init r=0, world=4 2022-08-17T12:19:55.5170918Z dist init r=2, world=4 2022-08-17T12:19:55.5392011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:55.5493698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:55.5494308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:55.5494911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:55.5495650Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:55.5496401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:55.5497086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:55.5497836Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:55.5602457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:55.5603187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:55.5603848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:55.5604511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:55.8923475Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:55.8929619Z test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-08-17T12:19:55.8978003Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13294 2022-08-17T12:19:55.9016180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13295 2022-08-17T12:19:55.9045711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13296 2022-08-17T12:19:55.9074011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13297 2022-08-17T12:19:57.0266062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:57.0266681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:57.0268320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:57.0268951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:57.0299221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:57.0299872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:57.0303141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:57.0303784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:57.0586829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:57.0587354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:57.0588403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:57.0588930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:57.0812289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:57.0812917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:57.0815060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:57.0815697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:57.1884509Z dist init r=1, world=4 2022-08-17T12:19:57.1917981Z dist init r=0, world=4 2022-08-17T12:19:57.2175032Z dist init r=2, world=4 2022-08-17T12:19:57.2412433Z dist init r=3, world=4 2022-08-17T12:19:57.2521948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:57.2587359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:57.2688767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:57.2689492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:57.2690139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:57.2690902Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:57.2691466Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:57.2725058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:57.2797460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:57.2798206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:57.2798900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:57.2799559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:57.6115941Z skip: Need at least 2 CUDA devices (1.719s) 2022-08-17T12:19:57.6123688Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:19:57.6172190Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13350 2022-08-17T12:19:57.6209708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13351 2022-08-17T12:19:57.6238088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13352 2022-08-17T12:19:57.6267445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13353 2022-08-17T12:19:58.6099633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:58.6100299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:58.6101845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:58.6102518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:58.6275448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:58.6276110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:58.6277135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:58.6277783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:58.6450168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:58.6450867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:58.6451875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:58.6452546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:58.6883367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:19:58.6883934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:19:58.6886681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:19:58.6887306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:19:58.7715673Z dist init r=2, world=4 2022-08-17T12:19:58.7875007Z dist init r=3, world=4 2022-08-17T12:19:58.8055677Z dist init r=1, world=4 2022-08-17T12:19:58.8473932Z dist init r=0, world=4 2022-08-17T12:19:58.8729731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:19:58.8832154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:19:58.8833112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:19:58.8834270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:58.8835005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:19:58.8835926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:58.8836933Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:58.8837663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:19:58.8840680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:19:58.8841365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:19:58.8841945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:19:58.8842792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:19:59.2308767Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:19:59.2316372Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:19:59.2365277Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13406 2022-08-17T12:19:59.2403972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13407 2022-08-17T12:19:59.2434134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13408 2022-08-17T12:19:59.2465135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13409 2022-08-17T12:20:00.2624143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:00.2624844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:00.2657669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:00.2658370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:00.3185121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:00.3185766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:00.3186815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:00.3187478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:00.3238395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:00.3239047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:00.3241607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:00.3242292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:00.3575589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:00.3576233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:00.3577824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:00.3578463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:00.4435864Z dist init r=1, world=4 2022-08-17T12:20:00.4811478Z dist init r=3, world=4 2022-08-17T12:20:00.4888167Z dist init r=2, world=4 2022-08-17T12:20:00.5171767Z dist init r=0, world=4 2022-08-17T12:20:00.5553041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:00.5553460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:00.5654737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:00.5655294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:00.5656089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:00.5656831Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:00.5657516Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:00.5658475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:00.5663753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:00.5664356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:00.5664909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:00.5665452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:00.8504224Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:00.8512568Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:20:00.8560168Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13462 2022-08-17T12:20:00.8599065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13463 2022-08-17T12:20:00.8629853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13464 2022-08-17T12:20:00.8660303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13465 2022-08-17T12:20:01.9248903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:01.9249506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:01.9251910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:01.9252486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:01.9366873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:01.9367483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:01.9369703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:01.9370335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:01.9390586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:01.9391221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:01.9394386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:01.9395045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:01.9396644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:01.9397248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:01.9400759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:01.9401379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:02.0895487Z dist init r=0, world=4 2022-08-17T12:20:02.0996088Z dist init r=2, world=4 2022-08-17T12:20:02.1011967Z dist init r=1, world=4 2022-08-17T12:20:02.1020012Z dist init r=3, world=4 2022-08-17T12:20:02.1206363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:02.1230803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:02.1332766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:02.1333555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:02.1334236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:02.1334942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:02.1335496Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:02.1409489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:02.1440428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:02.1441007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:02.1441820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:02.1442387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:02.4700104Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:02.4707682Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:20:02.4757103Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13518 2022-08-17T12:20:02.4796369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13519 2022-08-17T12:20:02.4829440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13520 2022-08-17T12:20:02.4874852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13521 2022-08-17T12:20:03.4780675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:03.4781337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:03.4782734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:03.4783357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:03.5029364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:03.5030110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:03.5031526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:03.5032068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:03.5080919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:03.5081728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:03.5084415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:03.5085076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:03.5430297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:03.5431581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:03.5432787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:03.5433416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:03.6404054Z dist init r=3, world=4 2022-08-17T12:20:03.6689065Z dist init r=1, world=4 2022-08-17T12:20:03.6766208Z dist init r=2, world=4 2022-08-17T12:20:03.7008029Z dist init r=0, world=4 2022-08-17T12:20:03.7420438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:03.7521212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:03.7623734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:03.7624744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:03.7625526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:03.7626311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:03.7627075Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:03.7627922Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:03.7632808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:03.7633401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:03.7634015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:03.7636615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:04.0916056Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:20:04.0923868Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:20:04.0972649Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13574 2022-08-17T12:20:04.1010306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13575 2022-08-17T12:20:04.1040136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13576 2022-08-17T12:20:04.1070358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13577 2022-08-17T12:20:05.0861030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:05.0861687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:05.0862882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:05.0863544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:05.0886929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:05.0887836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:05.0889946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:05.0890604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:05.1101677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:05.1102354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:05.1103534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:05.1104183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:05.1426928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:05.1427588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:05.1428652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:05.1429286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:05.2463061Z dist init r=2, world=4 2022-08-17T12:20:05.2484365Z dist init r=0, world=4 2022-08-17T12:20:05.2684534Z dist init r=1, world=4 2022-08-17T12:20:05.2974732Z dist init r=3, world=4 2022-08-17T12:20:05.3083252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:05.3097933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:05.3098690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:05.3099103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:05.3099762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:05.3100315Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:05.3100871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:05.3185736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:05.3205450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:05.3206306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:05.3206883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:05.3207457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:05.6109412Z skip: Need at least 2 CUDA devices (1.519s) 2022-08-17T12:20:05.6116151Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:20:05.6163698Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13630 2022-08-17T12:20:05.6202529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13631 2022-08-17T12:20:05.6234054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13632 2022-08-17T12:20:05.6283690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13633 2022-08-17T12:20:06.6078100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:06.6078765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:06.6079814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:06.6080465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:06.6110553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:06.6111208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:06.6114198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:06.6114867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:06.6250684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:06.6251338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:06.6253373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:06.6254004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:06.6904484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:06.6905367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:06.6906706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:06.6907384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:06.7710411Z dist init r=0, world=4 2022-08-17T12:20:06.7715148Z dist init r=3, world=4 2022-08-17T12:20:06.8024476Z dist init r=1, world=4 2022-08-17T12:20:06.8476009Z dist init r=2, world=4 2022-08-17T12:20:06.8686640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:06.8786954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:06.8888699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:06.8889437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:06.8890489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:06.8891052Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:06.8891613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:06.8892158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:06.8996816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:06.8997220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:06.8997607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:06.8997958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:07.2323650Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:20:07.2331562Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-08-17T12:20:07.2380141Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13686 2022-08-17T12:20:07.2418313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13687 2022-08-17T12:20:07.2449213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13688 2022-08-17T12:20:07.2479609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13689 2022-08-17T12:20:08.2703152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:08.2703798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:08.2705907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:08.2706566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:08.2900645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:08.2901261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:08.2902604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:08.2903274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:08.2904117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:08.2904979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:08.2908097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:08.2908758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:08.2939729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:08.2940373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:08.2942658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:08.2943338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:08.4306759Z dist init r=0, world=4 2022-08-17T12:20:08.4519116Z dist init r=2, world=4 2022-08-17T12:20:08.4521448Z dist init r=1, world=4 2022-08-17T12:20:08.4527525Z dist init r=3, world=4 2022-08-17T12:20:08.4727292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:08.4736770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:08.4838814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:08.4839234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:08.4839873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:08.4840448Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:08.4840998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:08.4930714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:08.4945638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:08.4946363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:08.4946904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:08.4947458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:08.8519075Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:08.8526796Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-08-17T12:20:08.8576641Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13742 2022-08-17T12:20:08.8612404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13743 2022-08-17T12:20:08.8641178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13744 2022-08-17T12:20:08.8670804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13745 2022-08-17T12:20:09.8826347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:09.8826982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:09.8828215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:09.8828899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:09.8907421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:09.8908093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:09.8909693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:09.8910454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:09.8977258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:09.8977892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:09.8980079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:09.8980659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:09.9186255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:09.9186818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:09.9188380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:09.9188887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:10.0448335Z dist init r=1, world=4 2022-08-17T12:20:10.0514586Z dist init r=3, world=4 2022-08-17T12:20:10.0587390Z dist init r=2, world=4 2022-08-17T12:20:10.0790858Z dist init r=0, world=4 2022-08-17T12:20:10.1026883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:10.1027382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:10.1128228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:10.1131890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:10.1132768Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:10.1230877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:10.1231699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:10.1232373Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:10.1239537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:10.1240272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:10.1241114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:10.1241668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:10.4711534Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:10.4716152Z test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-08-17T12:20:10.4765307Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13798 2022-08-17T12:20:10.4804506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13799 2022-08-17T12:20:10.4835564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13800 2022-08-17T12:20:10.4866883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13801 2022-08-17T12:20:11.5095770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:11.5096429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:11.5098015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:11.5098872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:11.5236337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:11.5237017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:11.5239928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:11.5240614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:11.5402397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:11.5403002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:11.5403979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:11.5404528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:11.5501739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:11.5502305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:11.5504370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:11.5504904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:11.6701714Z dist init r=3, world=4 2022-08-17T12:20:11.6843113Z dist init r=2, world=4 2022-08-17T12:20:11.7023804Z dist init r=0, world=4 2022-08-17T12:20:11.7084962Z dist init r=1, world=4 2022-08-17T12:20:11.7354477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:11.7456043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:11.7456631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:11.7457215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:11.7457849Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:11.7458411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:11.7458966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:11.7459688Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:11.7564182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:11.7564876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:11.7565453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:11.7566156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:12.0906539Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:12.0920475Z test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-08-17T12:20:12.0968500Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13854 2022-08-17T12:20:12.1006634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13855 2022-08-17T12:20:12.1036530Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13856 2022-08-17T12:20:12.1066174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13857 2022-08-17T12:20:13.0998302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:13.0999066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:13.1000374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:13.1000990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:13.1319389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:13.1319945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:13.1321901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:13.1322426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:13.1556247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:13.1556981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:13.1558648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:13.1559387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:13.1767698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:13.1768303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:13.1769532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:13.1770161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:13.2603273Z dist init r=1, world=4 2022-08-17T12:20:13.2922798Z dist init r=3, world=4 2022-08-17T12:20:13.3161817Z dist init r=2, world=4 2022-08-17T12:20:13.3378828Z dist init r=0, world=4 2022-08-17T12:20:13.3571214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:13.3672422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:13.3775166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:13.3776797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:13.3777603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:13.3778396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:13.3779264Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:13.3780103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:13.3782768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:13.3783900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:13.3784810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:13.3786830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:13.7107836Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:20:13.7121086Z test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-08-17T12:20:13.7168423Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13910 2022-08-17T12:20:13.7204273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13911 2022-08-17T12:20:13.7232865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13912 2022-08-17T12:20:13.7262040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13913 2022-08-17T12:20:14.7535710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:14.7536398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:14.7537812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:14.7538476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:14.7593866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:14.7594556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:14.7597269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:14.7597932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:14.7698821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:14.7699470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:14.7701754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:14.7702423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:14.7734258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:14.7734936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:14.7737446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:14.7738527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:14.9151666Z dist init r=2, world=4 2022-08-17T12:20:14.9194971Z dist init r=0, world=4 2022-08-17T12:20:14.9318375Z dist init r=3, world=4 2022-08-17T12:20:14.9333303Z dist init r=1, world=4 2022-08-17T12:20:14.9528676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:14.9706630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:14.9808639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:14.9809409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:14.9810261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:14.9810806Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:14.9811358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:14.9833750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:14.9916748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:14.9917355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:14.9917910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:14.9918688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:15.3301838Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:15.3302168Z 2022-08-17T12:20:15.3302687Z ---------------------------------------------------------------------- 2022-08-17T12:20:15.3303004Z Ran 35 tests in 56.947s 2022-08-17T12:20:15.3303128Z 2022-08-17T12:20:15.3303208Z OK (skipped=35) 2022-08-17T12:20:15.3303320Z 2022-08-17T12:20:15.3303413Z Generating XML reports... 2022-08-17T12:20:15.3387105Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220817121918.xml 2022-08-17T12:20:15.6651290Z Running distributed/fsdp/test_fsdp_overlap ... [2022-08-17 12:20:15.664741] 2022-08-17T12:20:15.6651905Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:20:15.664851] 2022-08-17T12:20:16.8065400Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-08-17T12:20:16.8078319Z 2022-08-17T12:20:16.8078462Z Running tests... 2022-08-17T12:20:16.8079016Z ---------------------------------------------------------------------- 2022-08-17T12:20:17.0703033Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13977 2022-08-17T12:20:18.0386734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:18.0387119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:18.0389756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:18.0390229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:18.2030482Z dist init r=0, world=1 2022-08-17T12:20:18.2040614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:18.2041205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:20:18.2044786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:18.4734632Z skip: Need at least 2 CUDA devices (1.665s) 2022-08-17T12:20:18.4742577Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:20:18.4743446Z 2022-08-17T12:20:18.4743675Z ---------------------------------------------------------------------- 2022-08-17T12:20:18.4743926Z Ran 2 tests in 1.666s 2022-08-17T12:20:18.4744049Z 2022-08-17T12:20:18.4744131Z OK (skipped=2) 2022-08-17T12:20:18.4744244Z 2022-08-17T12:20:18.4744332Z Generating XML reports... 2022-08-17T12:20:18.4777944Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220817122016.xml 2022-08-17T12:20:18.4780746Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220817122016.xml 2022-08-17T12:20:18.8067090Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-08-17 12:20:18.806282] 2022-08-17T12:20:18.8067936Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:20:18.806369] 2022-08-17T12:20:19.9395627Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap 2022-08-17T12:20:19.9407149Z 2022-08-17T12:20:19.9407354Z Running tests... 2022-08-17T12:20:19.9407961Z ---------------------------------------------------------------------- 2022-08-17T12:20:19.9418899Z test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-08-17T12:20:20.2066939Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14002 2022-08-17T12:20:20.2095067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14003 2022-08-17T12:20:20.2126450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14004 2022-08-17T12:20:20.2156811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14005 2022-08-17T12:20:21.2591106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:21.2591760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:21.2595977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:21.2596618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:21.2732453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:21.2733034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:21.2735423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:21.2736078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:21.3093464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:21.3094136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:21.3095356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:21.3096026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:21.3199441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:21.3200102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:21.3202271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:21.3202999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:21.4226933Z dist init r=2, world=4 2022-08-17T12:20:21.4337170Z dist init r=1, world=4 2022-08-17T12:20:21.4681882Z dist init r=0, world=4 2022-08-17T12:20:21.4770209Z dist init r=3, world=4 2022-08-17T12:20:21.5098202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:21.5098988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:21.5099709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:21.5100802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:21.5101377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:21.5101906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:21.5106152Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:21.5106778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:21.5206829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:21.5207472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:21.5208026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:21.5208572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:21.8200227Z skip: Need at least 2 CUDA devices (1.879s) 2022-08-17T12:20:21.8211395Z test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-08-17T12:20:21.8258929Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14058 2022-08-17T12:20:21.8296578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14059 2022-08-17T12:20:21.8325725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14060 2022-08-17T12:20:21.8355958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14061 2022-08-17T12:20:22.8521183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:22.8521829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:22.8523374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:22.8524010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:22.8750294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:22.8751029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:22.8752423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:22.8753123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:22.8820358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:22.8820980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:22.8823042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:22.8823664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:22.8862902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:22.8863735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:22.8866313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:22.8866970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:23.0094585Z dist init r=3, world=4 2022-08-17T12:20:23.0355004Z dist init r=2, world=4 2022-08-17T12:20:23.0400016Z dist init r=0, world=4 2022-08-17T12:20:23.0435355Z dist init r=1, world=4 2022-08-17T12:20:23.0748768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:23.0749500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:23.0750450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:23.0751115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:23.0751803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:23.0752215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:23.0752995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:23.0753549Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:23.0855778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:23.0856475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:23.0856870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:23.0857303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:23.4394854Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:23.4404245Z test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-08-17T12:20:23.4450707Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14114 2022-08-17T12:20:23.4488129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14115 2022-08-17T12:20:23.4517692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14116 2022-08-17T12:20:23.4547561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14117 2022-08-17T12:20:24.4622339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:24.4622980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:24.4624393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:24.4625020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:24.4844282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:24.4844943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:24.4847223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:24.4847892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:24.4916931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:24.4917538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:24.4919935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:24.4920598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:24.5010844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:24.5011509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:24.5014314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:24.5014975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:24.6194651Z dist init r=0, world=4 2022-08-17T12:20:24.6444811Z dist init r=2, world=4 2022-08-17T12:20:24.6497916Z dist init r=1, world=4 2022-08-17T12:20:24.6588607Z dist init r=3, world=4 2022-08-17T12:20:24.6757262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:24.6858569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:24.6859104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:24.6859746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:24.6860417Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:24.6860959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:24.6861509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:24.6862057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:24.6966654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:24.6967340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:24.6967898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:24.6968462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:25.0586338Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:25.0595663Z test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-08-17T12:20:25.0646691Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14170 2022-08-17T12:20:25.0683764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14171 2022-08-17T12:20:25.0712685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14172 2022-08-17T12:20:25.0741907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14173 2022-08-17T12:20:26.0900926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:26.0901612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:26.0903255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:26.0903905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:26.1071134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:26.1071823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:26.1073538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:26.1074411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:26.1121245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:26.1121968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:26.1124401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:26.1125095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:26.1195265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:26.1195953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:26.1199146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:26.1199824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:26.2472513Z dist init r=3, world=4 2022-08-17T12:20:26.2662929Z dist init r=1, world=4 2022-08-17T12:20:26.2693792Z dist init r=2, world=4 2022-08-17T12:20:26.2780682Z dist init r=0, world=4 2022-08-17T12:20:26.3178453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:26.3278555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:26.3380708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:26.3381848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:26.3382534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:26.3383105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:26.3383671Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:26.3386949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:26.3392530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:26.3393414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:26.3394166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:26.3396149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:26.6782338Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:26.6782555Z 2022-08-17T12:20:26.6782970Z ---------------------------------------------------------------------- 2022-08-17T12:20:26.6783252Z Ran 4 tests in 6.737s 2022-08-17T12:20:26.6783371Z 2022-08-17T12:20:26.6783446Z OK (skipped=4) 2022-08-17T12:20:26.6783596Z 2022-08-17T12:20:26.6783706Z Generating XML reports... 2022-08-17T12:20:26.6823772Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220817122019.xml 2022-08-17T12:20:27.0039774Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-08-17 12:20:27.003549] 2022-08-17T12:20:27.0040383Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:20:27.003637] 2022-08-17T12:20:28.1295731Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-08-17T12:20:28.1308274Z 2022-08-17T12:20:28.1308737Z Running tests... 2022-08-17T12:20:28.1309165Z ---------------------------------------------------------------------- 2022-08-17T12:20:28.1315468Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-08-17T12:20:28.3819270Z Tests pure FP16 training, including when the parameter's dtype is ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.251s) 2022-08-17T12:20:28.3823745Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-08-17T12:20:28.3919548Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14237 2022-08-17T12:20:28.3948542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14238 2022-08-17T12:20:28.3978002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14239 2022-08-17T12:20:28.4009085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14240 2022-08-17T12:20:29.4633207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:29.4634208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:29.4635713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:29.4636344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:29.5067056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:29.5067671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:29.5068380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:29.5068944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:29.5069586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:29.5070189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:29.5091836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:29.5092547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:29.5146349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:29.5146999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:29.5149457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:29.5150195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:29.6229781Z dist init r=0, world=4 2022-08-17T12:20:29.6649986Z dist init r=2, world=4 2022-08-17T12:20:29.6739003Z dist init r=3, world=4 2022-08-17T12:20:29.6825449Z dist init r=1, world=4 2022-08-17T12:20:29.7060530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:29.7144312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:29.7145004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:29.7145694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:29.7147634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:29.7148216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:29.7148954Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:29.7164976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:29.7255485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:29.7255979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:29.7256545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:29.7257073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:30.0055608Z skip: Need at least 2 CUDA devices (1.624s) 2022-08-17T12:20:30.0055786Z 2022-08-17T12:20:30.0056138Z ---------------------------------------------------------------------- 2022-08-17T12:20:30.0056414Z Ran 2 tests in 1.875s 2022-08-17T12:20:30.0056537Z 2022-08-17T12:20:30.0056653Z OK (skipped=2) 2022-08-17T12:20:30.0056778Z 2022-08-17T12:20:30.0056868Z Generating XML reports... 2022-08-17T12:20:30.0091415Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220817122028.xml 2022-08-17T12:20:30.3241336Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-08-17 12:20:30.323730] 2022-08-17T12:20:30.3241983Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:20:30.323837] 2022-08-17T12:20:31.4712867Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler 2022-08-17T12:20:31.4728393Z 2022-08-17T12:20:31.4728520Z Running tests... 2022-08-17T12:20:31.4729124Z ---------------------------------------------------------------------- 2022-08-17T12:20:31.4740560Z test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s) 2022-08-17T12:20:31.4748872Z test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s) 2022-08-17T12:20:31.4769851Z test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s) 2022-08-17T12:20:31.7464305Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14304 2022-08-17T12:20:31.7491814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14305 2022-08-17T12:20:31.7520286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14306 2022-08-17T12:20:31.7552648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14307 2022-08-17T12:20:32.8165708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:32.8166516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:32.8169297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:32.8169929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:32.8768103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:32.8768771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:32.8771512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:32.8772206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:32.8961378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:32.8962401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:32.8963858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:32.8964560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:32.9183401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:32.9183922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:32.9186637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:32.9187177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:32.9737157Z dist init r=3, world=4 2022-08-17T12:20:33.0337336Z dist init r=1, world=4 2022-08-17T12:20:33.0519520Z dist init r=2, world=4 2022-08-17T12:20:33.0744644Z dist init r=0, world=4 2022-08-17T12:20:33.0952965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:33.1054959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:33.1135207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:33.1135930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:33.1137041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:33.1137617Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:33.1156306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:33.1157629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:33.1243500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:33.1244102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:33.1244684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:33.1245265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:33.4597939Z skip: Need at least 2 CUDA devices (1.983s) 2022-08-17T12:20:33.4654518Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14360 2022-08-17T12:20:33.4691962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14361 2022-08-17T12:20:33.4721766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14362 2022-08-17T12:20:33.4751500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14363 2022-08-17T12:20:34.5220317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:34.5220975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:34.5223380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:34.5224065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:34.5291815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:34.5292431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:34.5296625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:34.5297321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:34.5461548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:34.5462236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:34.5465279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:34.5465901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:34.5592234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:34.5592943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:34.5596875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:34.5597470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:34.6796149Z dist init r=3, world=4 2022-08-17T12:20:34.6881635Z dist init r=2, world=4 2022-08-17T12:20:34.7033481Z dist init r=0, world=4 2022-08-17T12:20:34.7165757Z dist init r=1, world=4 2022-08-17T12:20:34.7308860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:34.7496440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:34.7599081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:34.7599965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:34.7600534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:34.7601534Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:34.7602200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:34.7612926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:34.7708570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:34.7709182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:34.7709574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:34.7710200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:35.0792286Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:35.0848594Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14416 2022-08-17T12:20:35.0886056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14417 2022-08-17T12:20:35.0915824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14418 2022-08-17T12:20:35.0946337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14419 2022-08-17T12:20:36.0944641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:36.0945325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:36.0947110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:36.0947762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:36.0987201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:36.0987860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:36.0991162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:36.0991804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:36.1103446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:36.1104085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:36.1107221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:36.1107863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:36.1454467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:36.1455083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:36.1457946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:36.1458555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:36.2533208Z dist init r=0, world=4 2022-08-17T12:20:36.2597322Z dist init r=1, world=4 2022-08-17T12:20:36.2695278Z dist init r=2, world=4 2022-08-17T12:20:36.3016458Z dist init r=3, world=4 2022-08-17T12:20:36.3210576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:36.3311552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:36.3412228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:36.3413447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:36.3414377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:36.3415262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:36.3415879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:36.3416444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:36.3422670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:36.3423384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:36.3424081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:36.3424987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:36.6986800Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:36.7043306Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14472 2022-08-17T12:20:36.7080789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14473 2022-08-17T12:20:36.7109362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14474 2022-08-17T12:20:36.7138286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14475 2022-08-17T12:20:37.7521973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:37.7522912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:37.7524824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:37.7525451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:37.7527368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:37.7527997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:37.7533020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:37.7533695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:37.7585818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:37.7586480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:37.7589976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:37.7590629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:37.7661767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:37.7662422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:37.7665973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:37.7666653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:37.9121715Z dist init r=1, world=4 2022-08-17T12:20:37.9126088Z dist init r=3, world=4 2022-08-17T12:20:37.9184940Z dist init r=2, world=4 2022-08-17T12:20:37.9223399Z dist init r=0, world=4 2022-08-17T12:20:37.9535044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:37.9635344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:37.9739564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:37.9740120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:37.9740786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:37.9741664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:37.9742291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:37.9742826Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:37.9747223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:37.9747921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:37.9748565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:37.9752166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:38.3177550Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:38.3235339Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14528 2022-08-17T12:20:38.3272187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14529 2022-08-17T12:20:38.3300620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14530 2022-08-17T12:20:38.3329795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14531 2022-08-17T12:20:39.3300561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:39.3301239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:39.3303461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:39.3304073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:39.3628277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:39.3628935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:39.3630708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:39.3631379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:39.3675249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:39.3676190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:39.3679510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:39.3680149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:39.3777559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:39.3778270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:39.3782153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:39.3782876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:39.4885324Z dist init r=1, world=4 2022-08-17T12:20:39.5236375Z dist init r=2, world=4 2022-08-17T12:20:39.5272535Z dist init r=3, world=4 2022-08-17T12:20:39.5365075Z dist init r=0, world=4 2022-08-17T12:20:39.5547625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:39.5586822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:39.5688330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:39.5688833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:39.5689474Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:39.5690031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:39.5690600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:39.5750826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:39.5797155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:39.5797661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:39.5798221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:39.5798787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:39.9370134Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:39.9426114Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14584 2022-08-17T12:20:39.9463315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14585 2022-08-17T12:20:39.9491882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14586 2022-08-17T12:20:39.9521177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14587 2022-08-17T12:20:40.9740629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:40.9741280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:40.9742948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:40.9743548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:41.0053075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:41.0053614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:41.0056262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:41.0057021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:41.0099257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:41.0099867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:41.0104392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:41.0105093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:41.0272638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:41.0273385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:41.0276731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:41.0277422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:41.1323870Z dist init r=1, world=4 2022-08-17T12:20:41.1709957Z dist init r=3, world=4 2022-08-17T12:20:41.1799146Z dist init r=2, world=4 2022-08-17T12:20:41.1857806Z dist init r=0, world=4 2022-08-17T12:20:41.2139825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:41.2342604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:41.2443978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:41.2444621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:41.2445291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:41.2446426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:41.2447216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:41.2448011Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:41.2453046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:41.2453445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:41.2453949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:41.2454695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:41.5560887Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:41.5616885Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14640 2022-08-17T12:20:41.5653559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14641 2022-08-17T12:20:41.5681547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14642 2022-08-17T12:20:41.5711146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14643 2022-08-17T12:20:42.5939678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:42.5940383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:42.5942606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:42.5943268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:42.6107450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:42.6108162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:42.6110036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:42.6110670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:42.6287931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:42.6288657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:42.6290933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:42.6291631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:42.6294792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:42.6295476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:42.6300641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:42.6301303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:42.7545066Z dist init r=1, world=4 2022-08-17T12:20:42.7703004Z dist init r=0, world=4 2022-08-17T12:20:42.7889585Z dist init r=3, world=4 2022-08-17T12:20:42.7893147Z dist init r=2, world=4 2022-08-17T12:20:42.8161187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:42.8263162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:42.8263834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:42.8264509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:42.8265206Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:42.8265763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:42.8266309Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:42.8266836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:42.8370942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:42.8371790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:42.8372360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:42.8372730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:43.1755047Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:43.1811241Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14696 2022-08-17T12:20:43.1850461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14697 2022-08-17T12:20:43.1880845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14698 2022-08-17T12:20:43.1911334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14699 2022-08-17T12:20:44.1999311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:44.1999971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:44.2002012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:44.2002669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:44.2373302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:44.2373976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:44.2376943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:44.2377639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:44.2381643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:44.2382223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:44.2386684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:44.2387333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:44.2615024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:44.2615734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:44.2617954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:44.2618609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:44.3603837Z dist init r=2, world=4 2022-08-17T12:20:44.4029542Z dist init r=0, world=4 2022-08-17T12:20:44.4165728Z dist init r=3, world=4 2022-08-17T12:20:44.4194728Z dist init r=1, world=4 2022-08-17T12:20:44.4378829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:44.4580691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:44.4683088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:20:44.4684051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:44.4684793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:44.4685416Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:44.4686166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:20:44.4686689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:20:44.4789965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:20:44.4790613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:44.4791245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:44.4791800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:20:44.7950838Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:20:44.7951136Z 2022-08-17T12:20:44.7951623Z ---------------------------------------------------------------------- 2022-08-17T12:20:44.7952104Z Ran 11 tests in 13.322s 2022-08-17T12:20:44.7952393Z 2022-08-17T12:20:44.7952545Z OK (skipped=11) 2022-08-17T12:20:44.7952737Z 2022-08-17T12:20:44.7952908Z Generating XML reports... 2022-08-17T12:20:44.7991237Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220817122031.xml 2022-08-17T12:20:44.8002078Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220817122031.xml 2022-08-17T12:20:45.1213342Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-08-17 12:20:45.120889] 2022-08-17T12:20:45.1213963Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:20:45.120975] 2022-08-17T12:20:46.2538161Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-08-17T12:20:46.2554800Z 2022-08-17T12:20:46.2555234Z Running tests... 2022-08-17T12:20:46.2555662Z ---------------------------------------------------------------------- 2022-08-17T12:20:46.2575161Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:20:46.5232812Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14763 2022-08-17T12:20:46.5260627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14764 2022-08-17T12:20:47.5230836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:47.5231250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:47.5233852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:47.5234217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:47.5265078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:47.5265627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:47.5269066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:47.5269498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:47.6807590Z dist init r=0, world=2 2022-08-17T12:20:47.6816422Z dist init r=1, world=2 2022-08-17T12:20:47.7016995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:47.7017714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:47.7018390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:47.7019160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:47.7022921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:47.7023419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:48.0298595Z skip: Need at least 2 CUDA devices (1.774s) 2022-08-17T12:20:48.0318284Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:20:48.0365547Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14791 2022-08-17T12:20:48.0403416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14792 2022-08-17T12:20:49.0183070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:49.0183757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:49.0185276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:49.0185957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:49.0438962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:49.0439594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:49.0442698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:49.0443323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:49.1729336Z dist init r=1, world=2 2022-08-17T12:20:49.1970323Z dist init r=0, world=2 2022-08-17T12:20:49.2139718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:49.2140290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:49.2141052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:49.2141689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:49.2146697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:49.2147332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:49.5435639Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:20:49.5456995Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:20:49.5505884Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14819 2022-08-17T12:20:49.5543529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14820 2022-08-17T12:20:50.5322294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:50.5322727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:50.5325272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:50.5325657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:50.5335688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:50.5336257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:50.5340653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:50.5341031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:50.6883740Z dist init r=1, world=2 2022-08-17T12:20:50.6903480Z dist init r=0, world=2 2022-08-17T12:20:50.7114053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:50.7114741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:50.7115355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:50.7115911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:50.7220085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:50.7220637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:51.0591885Z skip: Need at least 2 CUDA devices (1.515s) 2022-08-17T12:20:51.0611896Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:20:51.0660828Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14847 2022-08-17T12:20:51.0698673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14848 2022-08-17T12:20:52.0461890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:52.0462375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:52.0463583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:52.0464135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:52.0512990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:52.0513381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:52.0516395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:52.0516783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:52.2027783Z dist init r=1, world=2 2022-08-17T12:20:52.2083601Z dist init r=0, world=2 2022-08-17T12:20:52.2237284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:52.2237702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:52.2238504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:52.2239188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:52.2243431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:52.2244016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:52.5733566Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:20:52.5752968Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:20:52.5802407Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14875 2022-08-17T12:20:52.5840973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14876 2022-08-17T12:20:53.5657661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:53.5658059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:53.5660380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:53.5660896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:53.5688756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:53.5689372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:53.5692689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:53.5693264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:53.7231330Z dist init r=0, world=2 2022-08-17T12:20:53.7271137Z dist init r=1, world=2 2022-08-17T12:20:53.7381927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:53.7382368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:53.7382981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:53.7383548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:53.7386652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:53.7387198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:53.9872978Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:20:53.9892719Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:20:53.9940425Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14903 2022-08-17T12:20:53.9977458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14904 2022-08-17T12:20:54.9757106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:54.9757501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:54.9759724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:54.9760267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:54.9766039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:54.9766495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:54.9770303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:54.9770810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:55.1337475Z dist init r=0, world=2 2022-08-17T12:20:55.1337943Z dist init r=1, world=2 2022-08-17T12:20:55.1547707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:55.1548272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:55.1549058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:55.1549897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:55.1553010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:55.1553546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:55.4009371Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:20:55.4029067Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:20:55.4079032Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14931 2022-08-17T12:20:55.4116168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14932 2022-08-17T12:20:56.4225514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:56.4225912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:56.4228242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:56.4228641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:56.4255313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:56.4255675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:56.4259443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:56.4259813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:56.5778110Z dist init r=1, world=2 2022-08-17T12:20:56.5815019Z dist init r=0, world=2 2022-08-17T12:20:56.5987829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:56.5988365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:56.5989195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:56.5990039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:56.5994672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:56.5995167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:56.9149596Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:20:56.9169437Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:20:56.9217040Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14959 2022-08-17T12:20:56.9254258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14960 2022-08-17T12:20:57.9145110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:57.9145528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:57.9148328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:57.9148707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:57.9418051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:57.9418664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:57.9421429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:57.9421813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:58.0677092Z dist init r=1, world=2 2022-08-17T12:20:58.0941682Z dist init r=0, world=2 2022-08-17T12:20:58.1087831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:58.1088325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:58.1088967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:58.1089514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:58.1094119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:58.1094660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:58.4285500Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:20:58.4306262Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:20:58.4356155Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14987 2022-08-17T12:20:58.4393644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14988 2022-08-17T12:20:59.4121109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:59.4121505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:59.4124247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:59.4124613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:59.4129959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:20:59.4130437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:20:59.4134319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:20:59.4134681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:20:59.5669438Z dist init r=0, world=2 2022-08-17T12:20:59.5680257Z dist init r=1, world=2 2022-08-17T12:20:59.5878887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:20:59.5879311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:20:59.5879956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:59.5880719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:20:59.5985122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:20:59.5985735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:20:59.9425034Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:20:59.9445052Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:20:59.9494376Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15015 2022-08-17T12:20:59.9530961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15016 2022-08-17T12:21:00.9446834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:00.9447248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:00.9449698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:00.9450078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:00.9638461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:00.9638829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:00.9641884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:00.9642255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:01.1015534Z dist init r=1, world=2 2022-08-17T12:21:01.1210454Z dist init r=0, world=2 2022-08-17T12:21:01.1325722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:01.1326658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:01.1327303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:01.1327863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:01.1332143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:01.1332707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:01.4564532Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:01.4584448Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:01.4633769Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15043 2022-08-17T12:21:01.4672919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15044 2022-08-17T12:21:02.4712657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:02.4713051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:02.4715983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:02.4716558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:02.4932836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:02.4933281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:02.4936548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:02.4936926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:02.6290035Z dist init r=1, world=2 2022-08-17T12:21:02.6497132Z dist init r=0, world=2 2022-08-17T12:21:02.6707787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:02.6708462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:02.6709087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:02.6709939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:02.6713231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:02.6713765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:02.9703984Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:02.9722471Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:02.9770319Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15071 2022-08-17T12:21:02.9808005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15072 2022-08-17T12:21:03.9580639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:03.9581036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:03.9583495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:03.9583885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:03.9846758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:03.9847179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:03.9849728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:03.9850170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:04.1141368Z dist init r=1, world=2 2022-08-17T12:21:04.1380325Z dist init r=0, world=2 2022-08-17T12:21:04.1552389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:04.1552969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:04.1553711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:04.1554380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:04.1558726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:04.1559320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:04.4842576Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:04.4863255Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:04.4912460Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15099 2022-08-17T12:21:04.4949626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15100 2022-08-17T12:21:05.4642876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:05.4643529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:05.4646307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:05.4646920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:05.4673103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:05.4673984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:05.4676693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:05.4677392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:05.6207129Z dist init r=0, world=2 2022-08-17T12:21:05.6219476Z dist init r=1, world=2 2022-08-17T12:21:05.6328711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:05.6329299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:05.6329959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:05.6330596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:05.6334935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:05.6335354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:05.8980275Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:21:05.8999279Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:05.9047330Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15127 2022-08-17T12:21:05.9085816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15128 2022-08-17T12:21:06.8899721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:06.8900396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:06.8902786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:06.8903379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:06.9140340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:06.9140895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:06.9143428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:06.9143995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:07.0483910Z dist init r=0, world=2 2022-08-17T12:21:07.0727520Z dist init r=1, world=2 2022-08-17T12:21:07.0896106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:07.0896692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:07.0897346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:07.0897910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:07.1002488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:07.1002854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:07.4115915Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:07.4135497Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:07.4184925Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15155 2022-08-17T12:21:07.4224019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15156 2022-08-17T12:21:08.4126404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:08.4126986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:08.4128181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:08.4128739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:08.4200585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:08.4201068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:08.4204309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:08.4204739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:08.5706505Z dist init r=1, world=2 2022-08-17T12:21:08.5756320Z dist init r=0, world=2 2022-08-17T12:21:08.5967620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:08.5968261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:08.5969181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:08.5969986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:08.5972843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:08.5973227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:08.9253872Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:08.9274226Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:08.9322226Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15183 2022-08-17T12:21:08.9361947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15184 2022-08-17T12:21:09.9183049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:09.9183717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:09.9185742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:09.9186394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:09.9609179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:09.9609825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:09.9612251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:09.9612841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:10.0756265Z dist init r=1, world=2 2022-08-17T12:21:10.1176885Z dist init r=0, world=2 2022-08-17T12:21:10.1386795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:10.1387286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:10.1387944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:10.1388775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:10.1393708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:10.1394166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:10.4392817Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:10.4412782Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:10.4461106Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15211 2022-08-17T12:21:10.4499406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15212 2022-08-17T12:21:11.4290298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:11.4293357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:11.4294094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:11.4294480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:11.4307121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:11.4307479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:11.4312096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:11.4312507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:11.5862996Z dist init r=1, world=2 2022-08-17T12:21:11.5887159Z dist init r=0, world=2 2022-08-17T12:21:11.6098618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:11.6099130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:11.6100058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:11.6100619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:11.6104946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:11.6105457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:11.9532868Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:11.9552584Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:11.9600314Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15239 2022-08-17T12:21:11.9640473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15240 2022-08-17T12:21:12.9338800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:12.9339295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:12.9342179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:12.9342728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:12.9779766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:12.9780726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:12.9782731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:12.9783382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:13.0892705Z dist init r=0, world=2 2022-08-17T12:21:13.1329810Z dist init r=1, world=2 2022-08-17T12:21:13.1503857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:13.1504492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:13.1505168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:13.1505721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:13.1609800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:13.1610182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:13.4671328Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:13.4691665Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:13.4739632Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15267 2022-08-17T12:21:13.4776952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15268 2022-08-17T12:21:14.4637666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:14.4638351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:14.4640569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:14.4641163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:14.4658023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:14.4658692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:14.4661666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:14.4662355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:14.6193353Z dist init r=0, world=2 2022-08-17T12:21:14.6193758Z dist init r=1, world=2 2022-08-17T12:21:14.6301628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:14.6303210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:14.6304292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:14.6403964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:14.6408605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:14.6409205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:14.8806828Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:21:14.8825859Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:14.8873292Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15295 2022-08-17T12:21:14.8909613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15296 2022-08-17T12:21:15.8830867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:15.8831245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:15.8833924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:15.8834302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:15.8871982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:15.8872647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:15.8876493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:15.8876933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:16.0405606Z dist init r=1, world=2 2022-08-17T12:21:16.0420186Z dist init r=0, world=2 2022-08-17T12:21:16.0614397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:16.0615176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:16.0615843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:16.0616401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:16.0620283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:16.0620809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:16.3941831Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:16.3961037Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:16.4009968Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15323 2022-08-17T12:21:16.4047907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15324 2022-08-17T12:21:17.3866149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:17.3866543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:17.3868904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:17.3869365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:17.4255243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:17.4255677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:17.4258609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:17.4258995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:17.5407372Z dist init r=0, world=2 2022-08-17T12:21:17.5803535Z dist init r=1, world=2 2022-08-17T12:21:17.5913022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:17.5913557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:17.5914181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:17.5914940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:17.6020569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:17.6021102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:17.9079995Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:17.9100277Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:17.9149519Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15351 2022-08-17T12:21:17.9187936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15352 2022-08-17T12:21:18.9109264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:18.9110006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:18.9113712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:18.9114363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:18.9498292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:18.9498839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:18.9501332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:18.9501900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:19.0676483Z dist init r=1, world=2 2022-08-17T12:21:19.1046305Z dist init r=0, world=2 2022-08-17T12:21:19.1255175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:19.1255826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:19.1256772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:19.1271897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:19.1272353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:19.1272711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:19.4218938Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:19.4239062Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:19.4288173Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15379 2022-08-17T12:21:19.4326321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15380 2022-08-17T12:21:20.4305613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:20.4306017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:20.4308014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:20.4308442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:20.4621406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:20.4622039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:20.4624777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:20.4625165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:20.5861442Z dist init r=1, world=2 2022-08-17T12:21:20.6160789Z dist init r=0, world=2 2022-08-17T12:21:20.6370781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:20.6371193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:20.6371847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:20.6372407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:20.6376473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:20.6376852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:20.9357134Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:20.9376914Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:20.9423784Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15407 2022-08-17T12:21:20.9460327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15408 2022-08-17T12:21:21.9225853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:21.9226475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:21.9228627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:21.9229290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:21.9265083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:21.9265739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:21.9269698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:21.9270300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:22.0792823Z dist init r=1, world=2 2022-08-17T12:21:22.0847214Z dist init r=0, world=2 2022-08-17T12:21:22.1058171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:22.1058803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:22.1059449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:22.1060012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:22.1164637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:22.1165197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:22.4495147Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:22.4552034Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15435 2022-08-17T12:21:22.4588297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15436 2022-08-17T12:21:23.4666291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:23.4666689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:23.4670027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:23.4670623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:23.4671497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:23.4671879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:23.4675613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:23.4676073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:23.6253978Z dist init r=0, world=2 2022-08-17T12:21:23.6261970Z dist init r=1, world=2 2022-08-17T12:21:23.6371602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:23.6372416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:23.6373131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:23.6373807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:23.6378074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:23.6378458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:23.9623159Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:23.9679121Z test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15463 2022-08-17T12:21:23.9718093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15464 2022-08-17T12:21:24.9422108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:24.9422786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:24.9424200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:24.9424823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:24.9505906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:24.9506593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:24.9510691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:24.9511281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:25.0981054Z dist init r=0, world=2 2022-08-17T12:21:25.1077751Z dist init r=1, world=2 2022-08-17T12:21:25.1187149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:25.1187795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:25.1188501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:25.1189063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:25.1293541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:25.1294391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:25.4750701Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:25.4803749Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15491 2022-08-17T12:21:25.4840410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15492 2022-08-17T12:21:26.4754572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:26.4755203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:26.4757257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:26.4757837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:26.4765130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:26.4765814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:26.4806058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:26.4806925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:26.6381654Z dist init r=1, world=2 2022-08-17T12:21:26.6393815Z dist init r=0, world=2 2022-08-17T12:21:26.6592053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:26.6592792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:26.6593765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:26.6594551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:26.6598930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:26.6599363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:26.9871081Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:21:26.9880301Z test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_both (__main__.TestFSDPStateDict) 2022-08-17T12:21:26.9931867Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15519 2022-08-17T12:21:26.9970466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15520 2022-08-17T12:21:27.9697025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:27.9697437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:27.9700100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:27.9700597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:27.9830865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:27.9831340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:27.9834875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:27.9835377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:28.1250637Z dist init r=0, world=2 2022-08-17T12:21:28.1371472Z dist init r=1, world=2 2022-08-17T12:21:28.1561133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:28.1561879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:28.1562516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:28.1563084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:28.1567369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:28.1567802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:28.4000646Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:21:28.4009507Z test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_first (__main__.TestFSDPStateDict) 2022-08-17T12:21:28.4058182Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15547 2022-08-17T12:21:28.4094549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15548 2022-08-17T12:21:29.3741423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:29.3742027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:29.3744342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:29.3744764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:29.3761232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:29.3761591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:29.3765766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:29.3766381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:29.5268745Z dist init r=0, world=2 2022-08-17T12:21:29.5292388Z dist init r=1, world=2 2022-08-17T12:21:29.5401547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:29.5402021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:29.5402668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:29.5403232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:29.5509306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:29.5510046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:29.8123834Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:21:29.8132685Z test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_second (__main__.TestFSDPStateDict) 2022-08-17T12:21:29.8184236Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15575 2022-08-17T12:21:29.8220645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15576 2022-08-17T12:21:30.7943016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:30.7943680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:30.7946387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:30.7946964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:30.8048560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:30.8049518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:30.8051697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:30.8052369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:30.9498093Z dist init r=0, world=2 2022-08-17T12:21:30.9602570Z dist init r=1, world=2 2022-08-17T12:21:30.9711356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:30.9712077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:30.9712719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:30.9713279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:30.9817665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:30.9818129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:31.3254157Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:31.3278428Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:31.3328412Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15603 2022-08-17T12:21:31.3366925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15604 2022-08-17T12:21:32.3448588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:32.3449003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:32.3451522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:32.3451906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:32.3655745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:32.3656126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:32.3659160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:32.3659544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:32.5006248Z dist init r=0, world=2 2022-08-17T12:21:32.5217731Z dist init r=1, world=2 2022-08-17T12:21:32.5326655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:32.5327386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:32.5328073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:32.5328635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:32.5432792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:32.5433369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:32.8398784Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:32.8420737Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:32.8469064Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15631 2022-08-17T12:21:32.8507259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15632 2022-08-17T12:21:33.8194454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:33.8194858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:33.8198006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:33.8198672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:33.8218469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:33.8219142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:33.8223079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:33.8223734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:33.9734021Z dist init r=1, world=2 2022-08-17T12:21:33.9754090Z dist init r=0, world=2 2022-08-17T12:21:33.9964174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:33.9964598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:33.9965369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:33.9966406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:33.9969430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:33.9969909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:34.2536567Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:21:34.2557786Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:34.2606091Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15659 2022-08-17T12:21:34.2643718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15660 2022-08-17T12:21:35.2378440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:35.2379139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:35.2382042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:35.2382689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:35.2383498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:35.2384082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:35.2388235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:35.2388860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:35.3932055Z dist init r=1, world=2 2022-08-17T12:21:35.3938272Z dist init r=0, world=2 2022-08-17T12:21:35.4140223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:35.4140696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:35.4141344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:35.4143933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:35.4147051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:35.4148760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:35.6673708Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:21:35.6695098Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:35.6742880Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15687 2022-08-17T12:21:35.6779502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15688 2022-08-17T12:21:36.6542987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:36.6543402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:36.6546773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:36.6547467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:36.6578236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:36.6581939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:36.6582822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:36.6583474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:36.8124935Z dist init r=0, world=2 2022-08-17T12:21:36.8163133Z dist init r=1, world=2 2022-08-17T12:21:36.8335057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:36.8335686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:36.8336409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:36.8336970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:36.8340277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:36.8340952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:37.0809174Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:21:37.0830514Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:37.0877451Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15715 2022-08-17T12:21:37.0913580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15716 2022-08-17T12:21:38.0800027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:38.0800656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:38.0802194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:38.0802833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:38.0832641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:38.0833548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:38.0837466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:38.0838076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:38.2363814Z dist init r=1, world=2 2022-08-17T12:21:38.2385594Z dist init r=0, world=2 2022-08-17T12:21:38.2573277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:38.2573908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:38.2574990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:38.2575581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:38.2578981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:38.2579446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:38.5944981Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:21:38.5966816Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:38.6016066Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15743 2022-08-17T12:21:38.6053182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15744 2022-08-17T12:21:39.5947326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:39.5950054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:39.5950568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:39.5950944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:39.5996269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:39.5996634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:39.6000201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:39.6000602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:39.7520434Z dist init r=1, world=2 2022-08-17T12:21:39.7541964Z dist init r=0, world=2 2022-08-17T12:21:39.7751570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:39.7752113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:39.7752886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:39.7753644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:39.7757451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:39.7757937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:40.1087128Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:40.1108946Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:40.1157222Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15771 2022-08-17T12:21:40.1194502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15772 2022-08-17T12:21:41.0982975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:41.0983621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:41.0985406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:41.0985989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:41.0993093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:41.0993755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:41.0997199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:41.0997831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:41.2535053Z dist init r=1, world=2 2022-08-17T12:21:41.2538149Z dist init r=0, world=2 2022-08-17T12:21:41.2748447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:41.2749233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:41.2749983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:41.2750687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:41.2855618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:41.2856241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:41.6231711Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:41.6253116Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:41.6301976Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15799 2022-08-17T12:21:41.6340662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15800 2022-08-17T12:21:42.6261592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:42.6261990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:42.6264984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:42.6265366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:42.6276784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:42.6277245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:42.6281151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:42.6281679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:42.7832269Z dist init r=1, world=2 2022-08-17T12:21:42.7841057Z dist init r=0, world=2 2022-08-17T12:21:42.8043078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:42.8043793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:42.8044684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:42.8045805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:42.8048682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:42.8049321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:43.1375920Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:43.1397114Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:43.1445716Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15827 2022-08-17T12:21:43.1484982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15828 2022-08-17T12:21:44.1352298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:44.1352716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:44.1355042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:44.1355782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:44.1762162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:44.1762540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:44.1765244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:44.1765638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:44.2891049Z dist init r=1, world=2 2022-08-17T12:21:44.3296906Z dist init r=0, world=2 2022-08-17T12:21:44.3504071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:44.3504712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:44.3505521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:44.3506094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:44.3510732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:44.3511336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:44.6514828Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:44.6536445Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:44.6584783Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15855 2022-08-17T12:21:44.6620848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15856 2022-08-17T12:21:45.6448856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:45.6449260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:45.6451532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:45.6451898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:45.6640152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:45.6640519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:45.6643532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:45.6643962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:45.7990926Z dist init r=1, world=2 2022-08-17T12:21:45.8210511Z dist init r=0, world=2 2022-08-17T12:21:45.8419685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:45.8420463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:45.8421358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:45.8422267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:45.8424739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:45.8425278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:46.1654545Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:46.1676639Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:46.1723760Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15883 2022-08-17T12:21:46.1761181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15884 2022-08-17T12:21:47.1480374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:47.1481042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:47.1483670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:47.1484354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:47.1498471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:47.1499179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:47.1503295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:47.1503917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:47.3029137Z dist init r=1, world=2 2022-08-17T12:21:47.3050372Z dist init r=0, world=2 2022-08-17T12:21:47.3259954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:47.3260561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:47.3261479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:47.3262222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:47.3265392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:47.3265951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:47.5790175Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:21:47.5812281Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:47.5859820Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15911 2022-08-17T12:21:47.5896879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15912 2022-08-17T12:21:48.5901408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:48.5901809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:48.5904552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:48.5905013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:48.6212992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:48.6216448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:48.6217212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:48.6217590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:48.7496426Z dist init r=1, world=2 2022-08-17T12:21:48.7780633Z dist init r=0, world=2 2022-08-17T12:21:48.7992243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:48.7993212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:48.7994145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:48.7994985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:48.7998412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:48.7999082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:49.0928305Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:49.0952041Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:49.1000446Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15939 2022-08-17T12:21:49.1040828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15940 2022-08-17T12:21:50.0847162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:50.0847553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:50.0849686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:50.0850248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:50.1280706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:50.1281163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:50.1284095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:50.1284558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:50.2417990Z dist init r=1, world=2 2022-08-17T12:21:50.2829163Z dist init r=0, world=2 2022-08-17T12:21:50.3039602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:50.3040204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:50.3040917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:50.3041468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:50.3045294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:50.3046092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:50.6074477Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:50.6098137Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:50.6145753Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15967 2022-08-17T12:21:50.6183702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15968 2022-08-17T12:21:51.6042106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:51.6042537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:51.6045161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:51.6045546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:51.6195896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:51.6196290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:51.6199358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:51.6199734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:51.7631013Z dist init r=0, world=2 2022-08-17T12:21:51.7772480Z dist init r=1, world=2 2022-08-17T12:21:51.7942322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:51.7942888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:51.7943548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:51.7944118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:51.8049193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:51.8049786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:52.1214509Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:52.1238342Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:52.1287496Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15995 2022-08-17T12:21:52.1326097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15996 2022-08-17T12:21:53.1194250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:53.1194645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:53.1196938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:53.1197471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:53.1229275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:53.1229701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:53.1233162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:53.1233749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:53.2742582Z dist init r=1, world=2 2022-08-17T12:21:53.2765486Z dist init r=0, world=2 2022-08-17T12:21:53.2975346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:53.2975919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:53.2976673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:53.2977541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:53.2980934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:53.2981541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:53.6357844Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:53.6384717Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:53.6434091Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16023 2022-08-17T12:21:53.6470427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16024 2022-08-17T12:21:54.6251350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:54.6251755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:54.6254391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:54.6254866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:54.6526750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:54.6527147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:54.6529575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:54.6530098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:54.7809245Z dist init r=0, world=2 2022-08-17T12:21:54.8061634Z dist init r=1, world=2 2022-08-17T12:21:54.8170501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:54.8171348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:54.8171992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:54.8172573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:54.8276692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:54.8277322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:55.1502885Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:55.1525989Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:55.1575087Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16051 2022-08-17T12:21:55.1612742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16052 2022-08-17T12:21:56.1491786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:56.1492732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:56.1494206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:56.1494828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:56.1736795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:56.1737415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:56.1741052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:56.1741676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:56.3040045Z dist init r=0, world=2 2022-08-17T12:21:56.3298169Z dist init r=1, world=2 2022-08-17T12:21:56.3408158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:56.3435317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:56.3436419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:56.3510502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:56.3515665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:56.3516233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:56.6644459Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:56.6667955Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:56.6717021Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16079 2022-08-17T12:21:56.6753964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16080 2022-08-17T12:21:57.6458825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:57.6459535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:57.6462770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:57.6463404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:57.6568735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:57.6569380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:57.6572295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:57.6572984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:57.7994170Z dist init r=0, world=2 2022-08-17T12:21:57.8111393Z dist init r=1, world=2 2022-08-17T12:21:57.8304039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:57.8304468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:57.8305121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:57.8305682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:57.8410022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:57.8410477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:58.0784248Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:21:58.0807576Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-08-17T12:21:58.0854541Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16107 2022-08-17T12:21:58.0890043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16108 2022-08-17T12:21:59.1039397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:59.1041566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:59.1042520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:21:59.1042988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:21:59.1043453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:59.1043831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:59.1046667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:21:59.1047060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:21:59.2605462Z dist init r=1, world=2 2022-08-17T12:21:59.2605806Z dist init r=0, world=2 2022-08-17T12:21:59.2815948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:21:59.2816431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:21:59.2817080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:59.2817721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:21:59.2822214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:21:59.2822695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:21:59.5923322Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:21:59.5947394Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-08-17T12:21:59.5997087Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16135 2022-08-17T12:21:59.6033914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16136 2022-08-17T12:22:00.5799969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:00.5800801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:00.5801939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:00.5802450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:00.5803160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:00.5803777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:00.5804336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:00.5804700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:00.7349686Z dist init r=1, world=2 2022-08-17T12:22:00.7362600Z dist init r=0, world=2 2022-08-17T12:22:00.7572899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:00.7573452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:00.7574083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:00.7574643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:00.7679673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:00.7680240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:01.1066149Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:01.1082789Z test_state_dict_rank0_offload_save_load_flow (__main__.TestFSDPStateDict) 2022-08-17T12:22:01.1132188Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16163 2022-08-17T12:22:01.1170373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16164 2022-08-17T12:22:02.1515686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:02.1516343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:02.1518581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:02.1519165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:02.1569787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:02.1570442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:02.1573188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:02.1573883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:02.3082158Z dist init r=1, world=2 2022-08-17T12:22:02.3120545Z dist init r=0, world=2 2022-08-17T12:22:02.3331532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:02.3332012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:02.3332763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:02.3333326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:02.3336783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:02.3337409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:02.6201057Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:02.6251354Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16191 2022-08-17T12:22:02.6287762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16192 2022-08-17T12:22:03.6104750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:03.6105235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:03.6108087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:03.6108485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:03.6137346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:03.6137945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:03.6141042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:03.6141545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:03.7649649Z dist init r=0, world=2 2022-08-17T12:22:03.7683429Z dist init r=1, world=2 2022-08-17T12:22:03.7793518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:03.7793980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:03.7794708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:03.7795308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:03.7899575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:03.7900160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:04.0317104Z skip: Need at least 2 CUDA devices (1.411s) 2022-08-17T12:22:04.0369773Z test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16219 2022-08-17T12:22:04.0406165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16220 2022-08-17T12:22:05.0158765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:05.0159152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:05.0161120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:05.0161570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:05.0163913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:05.0164261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:05.0168629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:05.0169045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:05.1725334Z dist init r=1, world=2 2022-08-17T12:22:05.1725677Z dist init r=0, world=2 2022-08-17T12:22:05.1834288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:05.1836223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:05.1837287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:05.1936922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:05.1942282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:05.1943007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:05.4435419Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:22:05.4488962Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16247 2022-08-17T12:22:05.4525233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16248 2022-08-17T12:22:06.4513821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:06.4514823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:06.4515662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:06.4516331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:06.4916872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:06.4917524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:06.4918856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:06.4919452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:06.6076544Z dist init r=0, world=2 2022-08-17T12:22:06.6477847Z dist init r=1, world=2 2022-08-17T12:22:06.6586387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:06.6587055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:06.6587892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:06.6588447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:06.6692591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:06.6693170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:06.9557876Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:22:06.9630375Z test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16275 2022-08-17T12:22:06.9667069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16276 2022-08-17T12:22:07.9559742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:07.9560437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:07.9563470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:07.9563945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:07.9674422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:07.9674813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:07.9678507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:07.9678896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:08.1117348Z dist init r=0, world=2 2022-08-17T12:22:08.1246195Z dist init r=1, world=2 2022-08-17T12:22:08.1355040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:08.1355528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:08.1356168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:08.1356728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:08.1461120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:08.1461733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:08.4698438Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:08.4771875Z test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16303 2022-08-17T12:22:08.4809605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16304 2022-08-17T12:22:09.4508627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:09.4509262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:09.4511018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:09.4511668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:09.4998140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:09.4998734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:09.5000839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:09.5001435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:09.6038732Z dist init r=1, world=2 2022-08-17T12:22:09.6532095Z dist init r=0, world=2 2022-08-17T12:22:09.6649670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:09.6650419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:09.6651229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:09.6651794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:09.6655077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:09.6655741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:09.9840384Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:09.9915057Z test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16331 2022-08-17T12:22:09.9952714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16332 2022-08-17T12:22:10.9647836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:10.9648497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:10.9649312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:10.9649898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:10.9650725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:10.9651323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:10.9652109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:10.9652727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:11.1192174Z dist init r=0, world=2 2022-08-17T12:22:11.1192539Z dist init r=1, world=2 2022-08-17T12:22:11.1301510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:11.1302321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:11.1303173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:11.1303915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:11.1408465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:11.1409067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:11.3982140Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:22:11.4037014Z test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16359 2022-08-17T12:22:11.4073149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16360 2022-08-17T12:22:12.3812423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:12.3812821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:12.3815346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:12.3815793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:12.3839298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:12.3839894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:12.3843533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:12.3843893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:12.5363574Z dist init r=1, world=2 2022-08-17T12:22:12.5384903Z dist init r=0, world=2 2022-08-17T12:22:12.5573436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:12.5574135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:12.5575025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:12.5575614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:12.5579299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:12.5580003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:12.8104255Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:22:12.8174409Z test_state_dict_with_ignored_modules_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16387 2022-08-17T12:22:12.8210360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16388 2022-08-17T12:22:13.8000480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:13.8000877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:13.8003701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:13.8004063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:13.8343693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:13.8344068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:13.8346895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:13.8347256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:13.9634279Z dist init r=1, world=2 2022-08-17T12:22:13.9884264Z dist init r=0, world=2 2022-08-17T12:22:14.0044461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:14.0045131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:14.0046184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:14.0046852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:14.0050737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:14.0051252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:14.3240458Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:14.3311590Z test_state_dict_with_ignored_modules_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16415 2022-08-17T12:22:14.3349231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16416 2022-08-17T12:22:15.3125419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:15.3126102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:15.3128359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:15.3128740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:15.3181087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:15.3181429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:15.3185383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:15.3185767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:15.4691146Z dist init r=1, world=2 2022-08-17T12:22:15.4752808Z dist init r=0, world=2 2022-08-17T12:22:15.4900656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:15.4901385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:15.4902272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:15.4902900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:15.4906670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:15.4907073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:15.7381222Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:22:15.7452347Z test_state_dict_with_ignored_modules_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16443 2022-08-17T12:22:15.7488052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16444 2022-08-17T12:22:16.7289656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:16.7290034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:16.7292469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:16.7292852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:16.7586168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:16.7586753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:16.7590013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:16.7590395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:16.8837199Z dist init r=1, world=2 2022-08-17T12:22:16.9140713Z dist init r=0, world=2 2022-08-17T12:22:16.9351022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:16.9351552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:16.9352187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:16.9352746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:16.9457502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:16.9458125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:17.2519656Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:17.2603147Z test_state_dict_with_ignored_modules_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16471 2022-08-17T12:22:17.2628364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16472 2022-08-17T12:22:18.2528312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:18.2528695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:18.2531092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:18.2531559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:18.2747896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:18.2748252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:18.2752202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:18.2752580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:18.4089961Z dist init r=1, world=2 2022-08-17T12:22:18.4307438Z dist init r=0, world=2 2022-08-17T12:22:18.4517436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:18.4518116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:18.4518880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:18.4519437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:18.4523384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:18.4524062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:18.7659545Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:18.7716351Z test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16499 2022-08-17T12:22:18.7754718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16500 2022-08-17T12:22:19.7559495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:19.7560161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:19.7562128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:19.7562631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:19.7643345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:19.7643825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:19.7647095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:19.7647593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:19.9112771Z dist init r=0, world=2 2022-08-17T12:22:19.9223736Z dist init r=1, world=2 2022-08-17T12:22:19.9332630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:19.9333497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:19.9334163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:19.9334882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:19.9439833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:19.9440405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:20.2786181Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:20.2786632Z 2022-08-17T12:22:20.2787157Z ---------------------------------------------------------------------- 2022-08-17T12:22:20.2787506Z Ran 63 tests in 94.023s 2022-08-17T12:22:20.2787629Z 2022-08-17T12:22:20.2787724Z OK (skipped=63) 2022-08-17T12:22:20.2787839Z 2022-08-17T12:22:20.2787916Z Generating XML reports... 2022-08-17T12:22:20.2898337Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220817122046.xml 2022-08-17T12:22:20.6131787Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-08-17 12:22:20.612764] 2022-08-17T12:22:20.6132466Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:22:20.612852] 2022-08-17T12:22:21.7582731Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-08-17T12:22:21.7601567Z 2022-08-17T12:22:21.7601979Z Running tests... 2022-08-17T12:22:21.7602401Z ---------------------------------------------------------------------- 2022-08-17T12:22:22.0254767Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16538 2022-08-17T12:22:22.0282545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16539 2022-08-17T12:22:22.9963347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:22.9964071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:22.9964772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:22.9965283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:22.9966029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:22.9966391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:22.9966868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:22.9967448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:23.1518689Z dist init r=1, world=2 2022-08-17T12:22:23.1519290Z dist init r=0, world=2 2022-08-17T12:22:23.1729921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:23.1730526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:23.1731158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:23.1731726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:23.1836637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:23.1837117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:23.4315132Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T12:22:23.4370628Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16566 2022-08-17T12:22:23.4406882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16567 2022-08-17T12:22:24.4265595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:24.4265991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:24.4268572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:24.4269139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:24.4279038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:24.4279422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:24.4283046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:24.4283427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:24.5849375Z dist init r=0, world=2 2022-08-17T12:22:24.5849582Z dist init r=1, world=2 2022-08-17T12:22:24.6059716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:24.6060179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:24.6060816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:24.6061359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:24.6065409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:24.6065906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:24.9438218Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:22:24.9454493Z test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) 2022-08-17T12:22:24.9503257Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16594 2022-08-17T12:22:24.9540903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16595 2022-08-17T12:22:25.9402495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:25.9402883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:25.9405153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:25.9405792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:25.9784396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:25.9784822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:25.9786445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:25.9786818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:26.0953461Z dist init r=0, world=2 2022-08-17T12:22:26.1340955Z dist init r=1, world=2 2022-08-17T12:22:26.1449661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:26.1450399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:26.1451524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:26.1452146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:26.1557141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:26.1557871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:26.4571921Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:26.4584457Z test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) 2022-08-17T12:22:26.4632243Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16622 2022-08-17T12:22:26.4670459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16623 2022-08-17T12:22:27.4418449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:27.4418841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:27.4420524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:27.4420909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:27.4560342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:27.4560713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:27.4563660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:27.4564034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:27.5953831Z dist init r=1, world=2 2022-08-17T12:22:27.6098255Z dist init r=0, world=2 2022-08-17T12:22:27.6308492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:27.6308911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:27.6309640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:27.6310200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:27.6414978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:27.9704880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:27.9705218Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:27.9717335Z test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) 2022-08-17T12:22:27.9769338Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16650 2022-08-17T12:22:27.9806590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16651 2022-08-17T12:22:28.9518002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:28.9518385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:28.9520772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:28.9521180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:29.0045038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:29.0045450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:29.0047634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:29.0048074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:29.1051835Z dist init r=0, world=2 2022-08-17T12:22:29.1572804Z dist init r=1, world=2 2022-08-17T12:22:29.1681190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:29.1681703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:29.1682531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:29.1683156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:29.1787883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:29.1788291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:29.4836714Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:29.4849615Z test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) 2022-08-17T12:22:29.4893539Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16678 2022-08-17T12:22:29.4927523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16679 2022-08-17T12:22:30.4588696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:30.4589076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:30.4591234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:30.4591702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:30.4624147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:30.4624496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:30.4627683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:30.4628059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:30.6129747Z dist init r=1, world=2 2022-08-17T12:22:30.6173974Z dist init r=0, world=2 2022-08-17T12:22:30.6383551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:30.6384018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:30.6384652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:30.6385494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:30.6389914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:30.6390415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:30.8959520Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:22:30.9022297Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16706 2022-08-17T12:22:30.9058725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16707 2022-08-17T12:22:31.8841211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:31.8841717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:31.8844263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:31.8844721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:31.8845378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:31.8845718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:31.8848899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:31.8849273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:32.0380282Z dist init r=0, world=2 2022-08-17T12:22:32.0391088Z dist init r=1, world=2 2022-08-17T12:22:32.0499633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:32.0500414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:32.0501508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:32.0502159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:32.0605773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:32.0606600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:32.3088626Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:22:32.3152398Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16734 2022-08-17T12:22:32.3188567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16735 2022-08-17T12:22:33.2960296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:33.2960963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:33.2962425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:33.2963046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:33.2976413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:33.2977095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:33.2980715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:33.2981339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:33.4538270Z dist init r=1, world=2 2022-08-17T12:22:33.4538645Z dist init r=0, world=2 2022-08-17T12:22:33.4748732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:33.4749537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:33.4750326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:33.4751197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:33.4754212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:33.4754587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:33.7218278Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:22:33.7282223Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16762 2022-08-17T12:22:33.7320618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16763 2022-08-17T12:22:34.7094931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:34.7095533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:34.7096710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:34.7097088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:34.7099329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:34.7099689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:34.7102991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:34.7103360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:34.8654827Z dist init r=1, world=2 2022-08-17T12:22:34.8657716Z dist init r=0, world=2 2022-08-17T12:22:34.8864503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:34.8865001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:34.8865648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:34.8866194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:34.8971238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:34.8971824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:35.2351216Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:35.2422868Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16790 2022-08-17T12:22:35.2461002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16791 2022-08-17T12:22:36.2409453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:36.2410112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:36.2411395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:36.2412028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:36.2715277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:36.2715843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:36.2717220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:36.2717752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:36.3958023Z dist init r=1, world=2 2022-08-17T12:22:36.4261776Z dist init r=0, world=2 2022-08-17T12:22:36.4469684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:36.4470388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:36.4471283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:36.4472148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:36.4475500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:36.4476272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:36.7493653Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:36.7556741Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16818 2022-08-17T12:22:36.7594604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16819 2022-08-17T12:22:37.7536721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:37.7537140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:37.7538305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:37.7538664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:37.7651111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:37.7651475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:37.7654671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:37.7655066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:37.9117463Z dist init r=0, world=2 2022-08-17T12:22:37.9203175Z dist init r=1, world=2 2022-08-17T12:22:37.9326671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:37.9327180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:37.9328323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:37.9328921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:37.9433060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:37.9433452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:38.2628163Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:38.2695450Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16846 2022-08-17T12:22:38.2733661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16847 2022-08-17T12:22:39.2465929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:39.2466614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:39.2468941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:39.2469676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:39.2691067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:39.2691778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:39.2693194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:39.2693890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:39.4026776Z dist init r=1, world=2 2022-08-17T12:22:39.4243031Z dist init r=0, world=2 2022-08-17T12:22:39.4438207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:39.4439133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:39.4440021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:39.4440802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:39.4443676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:39.4444333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:39.7764360Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:39.7828400Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16874 2022-08-17T12:22:39.7865882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16875 2022-08-17T12:22:40.7649187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:40.7649582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:40.7671953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:40.7672359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:40.8100623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:40.8101223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:40.8102312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:40.8102877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:40.9248141Z dist init r=0, world=2 2022-08-17T12:22:40.9646868Z dist init r=1, world=2 2022-08-17T12:22:40.9755904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:40.9756655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:40.9757295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:40.9757853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:40.9863161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:40.9863843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:41.2897809Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:41.2962849Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16902 2022-08-17T12:22:41.3002198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16903 2022-08-17T12:22:42.2853353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:42.2854004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:42.2855250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:42.2855925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:42.2911337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:42.2912005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:42.2915674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:42.2916343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:42.4425519Z dist init r=0, world=2 2022-08-17T12:22:42.4505290Z dist init r=1, world=2 2022-08-17T12:22:42.4634674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:42.4635241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:42.4636079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:42.4636724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:42.4641252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:42.4641711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:42.8034477Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:42.8097792Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16930 2022-08-17T12:22:42.8138095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16931 2022-08-17T12:22:43.7880863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:43.7881431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:43.7882789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:43.7883201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:43.8323568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:43.8324065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:43.8326359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:43.8326862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:43.9436695Z dist init r=0, world=2 2022-08-17T12:22:43.9874850Z dist init r=1, world=2 2022-08-17T12:22:44.0048070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:44.0048910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:44.0049555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:44.0050119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:44.0154005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:44.0154480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:44.3168264Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:44.3231290Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16958 2022-08-17T12:22:44.3268145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16959 2022-08-17T12:22:45.2961876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:45.2962386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:45.2963578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:45.2963959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:45.3140890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:45.3141391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:45.3142839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:45.3143377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:45.4500266Z dist init r=1, world=2 2022-08-17T12:22:45.4711227Z dist init r=0, world=2 2022-08-17T12:22:45.4921421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:45.4921902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:45.4922567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:45.4923122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:45.4926987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:45.4927702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:45.8303402Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:45.8366375Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16986 2022-08-17T12:22:45.8404139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16987 2022-08-17T12:22:46.8155855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:46.8156369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:46.8158120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:46.8158483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:46.8161242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:46.8161605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:46.8164629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:46.8164989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:46.9720921Z dist init r=0, world=2 2022-08-17T12:22:46.9808718Z dist init r=1, world=2 2022-08-17T12:22:46.9917729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:46.9918526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:46.9919251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:46.9919813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:47.0024434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:47.0025069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:47.3435034Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:47.3499921Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17014 2022-08-17T12:22:47.3538914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17015 2022-08-17T12:22:48.3155944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:48.3156655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:48.3157665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:48.3158325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:48.3307410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:48.3308062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:48.3310574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:48.3311232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:48.4690422Z dist init r=1, world=2 2022-08-17T12:22:48.4863276Z dist init r=0, world=2 2022-08-17T12:22:48.5073986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:48.5074706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:48.5075363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:48.5075948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:48.5079833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:48.5080229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:48.7570510Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:22:48.7634891Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17042 2022-08-17T12:22:48.7673374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17043 2022-08-17T12:22:49.7635413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:49.7636009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:49.7637294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:49.7637787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:49.8074893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:49.8075406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:49.8077218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:49.8077725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:49.9202524Z dist init r=0, world=2 2022-08-17T12:22:49.9630130Z dist init r=1, world=2 2022-08-17T12:22:49.9738373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:49.9738814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:49.9739460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:49.9740186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:49.9844896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:49.9845429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:50.2704640Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:22:50.2767491Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17070 2022-08-17T12:22:50.2805455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17071 2022-08-17T12:22:51.2561771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:51.2562431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:51.2563481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:51.2564155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:51.2570079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:51.2570748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:51.2574426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:51.2575086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:51.4139618Z dist init r=1, world=2 2022-08-17T12:22:51.4156810Z dist init r=0, world=2 2022-08-17T12:22:51.4367991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:51.4368723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:51.4369513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:51.4370390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:51.4374192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:51.4374628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:51.6836374Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:22:51.6898770Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17098 2022-08-17T12:22:51.6936040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17099 2022-08-17T12:22:52.6788927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:52.6789760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:52.6790819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:52.6791431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:52.6816160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:52.6816862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:52.6820318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:52.6820720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:52.8371152Z dist init r=0, world=2 2022-08-17T12:22:52.8394693Z dist init r=1, world=2 2022-08-17T12:22:52.8504346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:52.8505011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:52.8505626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:52.8506185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:52.8510155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:52.8510553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:53.0991491Z skip: Need at least 2 CUDA devices (1.415s) 2022-08-17T12:22:53.1055514Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17126 2022-08-17T12:22:53.1094884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17127 2022-08-17T12:22:54.0791144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:54.0791788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:54.0793186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:54.0793831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:54.0839778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:54.0840428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:54.0842941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:54.0843644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:54.2361303Z dist init r=1, world=2 2022-08-17T12:22:54.2412835Z dist init r=0, world=2 2022-08-17T12:22:54.2570921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:54.2571485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:54.2572148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:54.2572707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:54.2677202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:54.2677585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:54.5123947Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:22:54.5130560Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) 2022-08-17T12:22:54.5179682Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17154 2022-08-17T12:22:54.5217609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17155 2022-08-17T12:22:55.4861216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:55.4861712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:55.4863803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:55.4864218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:55.4930060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:55.4930465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:55.4933136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:55.4933512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:55.6421451Z dist init r=0, world=2 2022-08-17T12:22:55.6488271Z dist init r=1, world=2 2022-08-17T12:22:55.6597160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:55.6597799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:55.6598441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:55.6599004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:55.6703863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:55.6704434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:55.9248651Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:22:55.9319351Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17182 2022-08-17T12:22:55.9359201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17183 2022-08-17T12:22:56.9287958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:56.9288635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:56.9290604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:56.9291255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:56.9587315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:56.9587843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:56.9589116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:56.9589640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:57.0879499Z dist init r=0, world=2 2022-08-17T12:22:57.1163363Z dist init r=1, world=2 2022-08-17T12:22:57.1271633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:57.1272290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:57.1272951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:57.1273514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:57.1378585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:57.1379232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:57.4391278Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:57.4460028Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17210 2022-08-17T12:22:57.4499487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17211 2022-08-17T12:22:58.4290883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:58.4291589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:58.4292816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:58.4293486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:58.4627463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:58.4627962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:58.4629314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:58.4629893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:58.5855435Z dist init r=0, world=2 2022-08-17T12:22:58.6182792Z dist init r=1, world=2 2022-08-17T12:22:58.6367138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:22:58.6367821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:22:58.6368636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:58.6369186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:22:58.6474521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:22:58.6475077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:22:58.9531494Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:22:58.9600389Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17238 2022-08-17T12:22:58.9638482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17239 2022-08-17T12:22:59.9367583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:59.9368232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:59.9369878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:59.9370542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:22:59.9826264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:22:59.9826958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:22:59.9828022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:22:59.9828672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:00.0917231Z dist init r=0, world=2 2022-08-17T12:23:00.1366248Z dist init r=1, world=2 2022-08-17T12:23:00.1529204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:00.1529847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:00.1530669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:00.1531249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:00.1635052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:00.1635896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:00.4668991Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:23:00.4737969Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17266 2022-08-17T12:23:00.4776441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17267 2022-08-17T12:23:01.4472701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:01.4473120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:01.4475199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:01.4475563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:01.4568015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:01.4568519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:01.4571227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:01.4571735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:01.6013093Z dist init r=0, world=2 2022-08-17T12:23:01.6109105Z dist init r=1, world=2 2022-08-17T12:23:01.6222283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:01.6222808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:01.6223596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:01.6224170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:01.6328331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:01.6328800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:01.9807381Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:23:01.9876754Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17294 2022-08-17T12:23:01.9914897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17295 2022-08-17T12:23:02.9716051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:02.9716447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:02.9717774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:02.9718407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:02.9774609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:02.9775086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:02.9777834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:02.9778367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:03.1287249Z dist init r=0, world=2 2022-08-17T12:23:03.1355648Z dist init r=1, world=2 2022-08-17T12:23:03.1465180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:03.1465923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:03.1466740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:03.1467303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:03.1471440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:03.1471847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:03.3945287Z skip: Need at least 2 CUDA devices (1.414s) 2022-08-17T12:23:03.4013419Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17322 2022-08-17T12:23:03.4051713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17323 2022-08-17T12:23:04.3936757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:04.3937222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:04.3937823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:04.3938372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:04.3938998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:04.3939479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:04.3940024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:04.3940390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:04.5490598Z dist init r=1, world=2 2022-08-17T12:23:04.5504606Z dist init r=0, world=2 2022-08-17T12:23:04.5699929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:04.5700635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:04.5701450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:04.5702350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:04.5705379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:04.5705757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:04.8081512Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:04.8149748Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17350 2022-08-17T12:23:04.8185877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17351 2022-08-17T12:23:05.7921953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:05.7922603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:05.7924269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:05.7924942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:05.7963702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:05.7964085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:05.7967538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:05.7967953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:05.9510596Z dist init r=1, world=2 2022-08-17T12:23:05.9560374Z dist init r=0, world=2 2022-08-17T12:23:05.9720204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:05.9720855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:05.9721499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:05.9722060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:05.9726970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:05.9727583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:06.2216296Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:06.2284263Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17378 2022-08-17T12:23:06.2323051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17379 2022-08-17T12:23:07.2159318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:07.2159744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:07.2161928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:07.2162297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:07.2212853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:07.2213532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:07.2216905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:07.2217552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:07.3739953Z dist init r=0, world=2 2022-08-17T12:23:07.3800818Z dist init r=1, world=2 2022-08-17T12:23:07.3949636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:07.3950510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:07.3951175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:07.3951741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:07.4056150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:07.4056778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:07.7356112Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:23:07.7415371Z test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17406 2022-08-17T12:23:07.7453349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17407 2022-08-17T12:23:08.7404152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:08.7404841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:08.7406161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:08.7406844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:08.7467754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:08.7468174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:08.7470835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:08.7471351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:08.8968812Z dist init r=1, world=2 2022-08-17T12:23:08.9028966Z dist init r=0, world=2 2022-08-17T12:23:08.9178104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:08.9178638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:08.9179466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:08.9180031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:08.9183297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:08.9183721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:09.2484972Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:09.2549708Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17434 2022-08-17T12:23:09.2588202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17435 2022-08-17T12:23:10.2310682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:10.2311347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:10.2312610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:10.2313117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:10.2430053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:10.2430694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:10.2433179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:10.2433574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:10.3868803Z dist init r=1, world=2 2022-08-17T12:23:10.3996868Z dist init r=0, world=2 2022-08-17T12:23:10.4207731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:10.4208309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:10.4209043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:10.4209600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:10.4212975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:10.4213365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:10.7623171Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:23:10.7686833Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17462 2022-08-17T12:23:10.7726560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17463 2022-08-17T12:23:11.7530005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:11.7530628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:11.7532274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:11.7532921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:11.7645695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:11.7646583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:11.7649412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:11.7650056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:11.9101351Z dist init r=1, world=2 2022-08-17T12:23:11.9210851Z dist init r=0, world=2 2022-08-17T12:23:11.9421680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:11.9422325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:11.9422977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:11.9423542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:11.9528182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:11.9528741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:12.2757204Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:12.2822643Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17490 2022-08-17T12:23:12.2861125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17491 2022-08-17T12:23:13.2714199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:13.2714859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:13.2716175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:13.2716582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:13.3031160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:13.3031759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:13.3033372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:13.3034013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:13.4309512Z dist init r=1, world=2 2022-08-17T12:23:13.4586415Z dist init r=0, world=2 2022-08-17T12:23:13.4720443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:13.4721078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:13.4722036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:13.4722866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:13.4726729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:13.4727122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:13.7895906Z skip: Need at least 2 CUDA devices (1.514s) 2022-08-17T12:23:13.7958887Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17518 2022-08-17T12:23:13.7995458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17519 2022-08-17T12:23:14.7780470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:14.7780993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:14.7782584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:14.7783246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:14.8132187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:14.8132733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:14.8134123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:14.8134650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:14.9357685Z dist init r=0, world=2 2022-08-17T12:23:14.9661951Z dist init r=1, world=2 2022-08-17T12:23:14.9868836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:14.9869437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:14.9870282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:14.9870832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:14.9974817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:14.9975300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:15.3026727Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:15.3091473Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17546 2022-08-17T12:23:15.3129097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17547 2022-08-17T12:23:16.3056479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:16.3057026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:16.3057717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:16.3058094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:16.3149764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:16.3150121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:16.3152471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:16.3152840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:16.4652130Z dist init r=0, world=2 2022-08-17T12:23:16.4723765Z dist init r=1, world=2 2022-08-17T12:23:16.4861116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:16.4861541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:16.4862202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:16.4862754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:16.4968300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:16.4968829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:16.8163619Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:16.8228281Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17574 2022-08-17T12:23:16.8266947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17575 2022-08-17T12:23:17.8544736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:17.8545303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:17.8546314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:17.8546689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:17.8689132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:17.8689503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:17.8691463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:17.8691833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:18.0193738Z dist init r=1, world=2 2022-08-17T12:23:18.0304185Z dist init r=0, world=2 2022-08-17T12:23:18.0503123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:18.0503798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:18.0504457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:18.0505198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:18.0609085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:18.0609506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:18.3299212Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:18.3362620Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17602 2022-08-17T12:23:18.3400246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17603 2022-08-17T12:23:19.3373370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:19.3374027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:19.3375679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:19.3376336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:19.3389958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:19.3390357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:19.3393179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:19.3393559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:19.4949782Z dist init r=0, world=2 2022-08-17T12:23:19.4976253Z dist init r=1, world=2 2022-08-17T12:23:19.5160332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:19.5160829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:19.5161487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:19.5162034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:19.5166236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:19.5166803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:19.8431144Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:19.8495046Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17630 2022-08-17T12:23:19.8532610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17631 2022-08-17T12:23:20.8324041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:20.8324732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:20.8325528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:20.8326289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:20.8327088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:20.8327698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:20.8328526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:20.8329165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:20.9882657Z dist init r=1, world=2 2022-08-17T12:23:20.9889508Z dist init r=0, world=2 2022-08-17T12:23:21.0100378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:21.0100811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:21.0101485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:21.0102046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:21.0207034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:21.0207666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:21.3564180Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:21.3629269Z test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17658 2022-08-17T12:23:21.3670303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17659 2022-08-17T12:23:22.3359615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:22.3360095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:22.3360651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:22.3361070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:22.3362264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:22.3362914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:22.3363772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:22.3364418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:22.4917077Z dist init r=1, world=2 2022-08-17T12:23:22.4920847Z dist init r=0, world=2 2022-08-17T12:23:22.5131948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:22.5132369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:22.5133010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:22.5133567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:22.5237875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:22.5238386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:22.7700195Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:22.7760077Z test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17686 2022-08-17T12:23:22.7796717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17687 2022-08-17T12:23:23.7686761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:23.7687198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:23.7689260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:23.7689805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:23.7769649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:23.7770391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:23.7772073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:23.7772504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:23.9259819Z dist init r=1, world=2 2022-08-17T12:23:23.9332500Z dist init r=0, world=2 2022-08-17T12:23:23.9468782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:23.9469453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:23.9470209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:23.9470894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:23.9475138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:23.9475568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:24.2829111Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:24.2884437Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17714 2022-08-17T12:23:24.2921135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17715 2022-08-17T12:23:25.2612850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:25.2613373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:25.2614488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:25.2614956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:25.2633326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:25.2633672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:25.2636792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:25.2637165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:25.4176021Z dist init r=0, world=2 2022-08-17T12:23:25.4217430Z dist init r=1, world=2 2022-08-17T12:23:25.4386566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:25.4387241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:25.4387955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:25.4388516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:25.4392805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:25.4393398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:25.6952269Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:23:25.7007353Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17742 2022-08-17T12:23:25.7045435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17743 2022-08-17T12:23:26.7095184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:26.7095758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:26.7096940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:26.7097318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:26.7239464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:26.7239830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:26.7242437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:26.7242888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:26.8665725Z dist init r=1, world=2 2022-08-17T12:23:26.8805111Z dist init r=0, world=2 2022-08-17T12:23:26.8975584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:26.8976208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:26.8977288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:26.8977911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:26.8982005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:26.8982630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:27.2077063Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:27.2134724Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17770 2022-08-17T12:23:27.2173413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17771 2022-08-17T12:23:28.1837357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:28.1838141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:28.1838958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:28.1839575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:28.1840331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:28.1840903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:28.1841722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:28.1842337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:28.3403465Z dist init r=0, world=2 2022-08-17T12:23:28.3408905Z dist init r=1, world=2 2022-08-17T12:23:28.3612919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:28.3613354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:28.3614016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:28.3614568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:28.3718496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:28.3719327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:28.6201741Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:23:28.6258346Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17798 2022-08-17T12:23:28.6296171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17799 2022-08-17T12:23:29.6107357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:29.6107847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:29.6109695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:29.6110248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:29.6426444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:29.6426930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:29.6429183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:29.6429620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:29.7682674Z dist init r=0, world=2 2022-08-17T12:23:29.7992481Z dist init r=1, world=2 2022-08-17T12:23:29.8194637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:29.8195074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:29.8195739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:29.8196477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:29.8301041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:29.8301633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:30.1328007Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:30.1384278Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17826 2022-08-17T12:23:30.1422475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17827 2022-08-17T12:23:31.1087887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:31.1088540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:31.1089935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:31.1090570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:31.1118900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:31.1119289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:31.1122206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:31.1122663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:31.2634700Z dist init r=0, world=2 2022-08-17T12:23:31.2668962Z dist init r=1, world=2 2022-08-17T12:23:31.2778546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:31.2779348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:31.2780115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:31.2780677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:31.2784128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:31.2784521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:31.5452823Z skip: Need at least 2 CUDA devices (1.412s) 2022-08-17T12:23:31.5508907Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17854 2022-08-17T12:23:31.5545885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17855 2022-08-17T12:23:32.5272883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:32.5273527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:32.5275014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:32.5275614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:32.5776126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:32.5776559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:32.5778861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:32.5779293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:32.6796950Z dist init r=0, world=2 2022-08-17T12:23:32.7284807Z dist init r=1, world=2 2022-08-17T12:23:32.7408942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:32.7409802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:32.7410218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:32.7410750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:32.7515036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:32.7515496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:33.0577954Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:33.0633978Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17882 2022-08-17T12:23:33.0671700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17883 2022-08-17T12:23:34.0590338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:34.0590975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:34.0592553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:34.0593192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:34.1098944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:34.1100011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:34.1101095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:34.1101816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:34.2142683Z dist init r=0, world=2 2022-08-17T12:23:34.2650795Z dist init r=1, world=2 2022-08-17T12:23:34.2759144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:34.2759791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:34.2760590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:34.2761207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:34.2865859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:34.2866463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:34.5706360Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:34.5763009Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17910 2022-08-17T12:23:34.5801418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17911 2022-08-17T12:23:35.5563265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:35.5563740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:35.5566110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:35.5566532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:35.5577104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:35.5577464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:35.5580912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:35.5581293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:35.7124388Z dist init r=1, world=2 2022-08-17T12:23:35.7138280Z dist init r=0, world=2 2022-08-17T12:23:35.7333553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:35.7334055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:35.7334762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:35.7335378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:35.7438386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:35.7438859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:36.0831378Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:36.0886581Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17938 2022-08-17T12:23:36.0923968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17939 2022-08-17T12:23:37.1044864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:37.1045421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:37.1047276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:37.1047641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:37.1203660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:37.1204021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:37.1206918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:37.1207314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:37.2602175Z dist init r=1, world=2 2022-08-17T12:23:37.2742638Z dist init r=0, world=2 2022-08-17T12:23:37.2912674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:37.2913594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:37.2914242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:37.2914787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:37.2918350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:37.2918947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:37.5955521Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:37.6010875Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17966 2022-08-17T12:23:37.6048182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17967 2022-08-17T12:23:38.5926935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:38.5927447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:38.5928488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:38.5929019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:38.6307171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:38.6307755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:38.6309972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:38.6310531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:38.7494629Z dist init r=1, world=2 2022-08-17T12:23:38.7891562Z dist init r=0, world=2 2022-08-17T12:23:38.8101952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:38.8102420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:38.8103078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:38.8103903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:38.8108154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:38.8108919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:39.1079265Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:39.1137996Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17994 2022-08-17T12:23:39.1176697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17995 2022-08-17T12:23:40.0967511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:40.0967966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:40.0969049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:40.0969482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:40.1000392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:40.1000816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:40.1004172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:40.1004552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:40.2524798Z dist init r=0, world=2 2022-08-17T12:23:40.2553178Z dist init r=1, world=2 2022-08-17T12:23:40.2662481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:40.2663100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:40.2664169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:40.2664872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:40.2668633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:40.2669178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:40.5207051Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:40.5261365Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18022 2022-08-17T12:23:40.5297730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18023 2022-08-17T12:23:41.5101430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:41.5101992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:41.5102591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:41.5103028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:41.5104018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:41.5104448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:41.5104908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:41.5105353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:41.6695609Z dist init r=1, world=2 2022-08-17T12:23:41.6841469Z dist init r=0, world=2 2022-08-17T12:23:41.7006244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:41.7006911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:41.7007675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:41.7008232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:41.7012100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:41.7012489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:42.0333829Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:42.0390714Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18050 2022-08-17T12:23:42.0430381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18051 2022-08-17T12:23:43.0516311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:43.0516988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:43.0518123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:43.0518764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:43.0593493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:43.0593883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:43.0596484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:43.0596885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:43.2080965Z dist init r=1, world=2 2022-08-17T12:23:43.2159805Z dist init r=0, world=2 2022-08-17T12:23:43.2290152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:43.2290653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:43.2291401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:43.2292285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:43.2295955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:43.2296338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:43.5460290Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:43.5520021Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18078 2022-08-17T12:23:43.5558546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18079 2022-08-17T12:23:44.5478752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:44.5479153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:44.5482410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:44.5482797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:44.5483486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:44.5483946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:44.5485359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:44.5485725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:44.7051134Z dist init r=1, world=2 2022-08-17T12:23:44.7063477Z dist init r=0, world=2 2022-08-17T12:23:44.7261260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:44.7261924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:44.7262570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:44.7263132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:44.7379442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:44.7379985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:45.0593513Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:45.0649947Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18106 2022-08-17T12:23:45.0688744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18107 2022-08-17T12:23:46.0385102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:46.0385543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:46.0387359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:46.0387799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:46.0505670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:46.0506051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:46.0508618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:46.0509079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:46.1926603Z dist init r=1, world=2 2022-08-17T12:23:46.2047233Z dist init r=0, world=2 2022-08-17T12:23:46.2235839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:46.2236607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:46.2237265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:46.2237824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:46.2241999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:46.2242594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:46.4719464Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:46.4774055Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18134 2022-08-17T12:23:46.4810453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18135 2022-08-17T12:23:47.4719761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:47.4720404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:47.4721998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:47.4722648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:47.4815423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:47.4816076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:47.4818907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:47.4819592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:47.6272825Z dist init r=1, world=2 2022-08-17T12:23:47.6356900Z dist init r=0, world=2 2022-08-17T12:23:47.6566826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:47.6567711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:47.6568382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:47.6568942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:47.6572936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:47.6573620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:47.9843381Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:47.9904967Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18162 2022-08-17T12:23:47.9943031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18163 2022-08-17T12:23:48.9742810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:48.9743523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:48.9744385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:48.9745038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:48.9995564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:48.9996255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:48.9998804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:48.9999480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:49.1313510Z dist init r=1, world=2 2022-08-17T12:23:49.1550846Z dist init r=0, world=2 2022-08-17T12:23:49.1759932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:49.1760514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:49.1761390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:49.1761952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:49.1766142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:49.1767008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:49.4974594Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:49.5036723Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18190 2022-08-17T12:23:49.5074080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18191 2022-08-17T12:23:50.4978559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:50.4979170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:50.4980303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:50.4980712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:50.5330171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:50.5330580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:50.5333562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:50.5333964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:50.6560429Z dist init r=0, world=2 2022-08-17T12:23:50.6877019Z dist init r=1, world=2 2022-08-17T12:23:50.6986280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:50.6987028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:50.6987742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:50.6988303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:50.7092883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:50.7093514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:51.0108578Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:51.0169117Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18218 2022-08-17T12:23:51.0205293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18219 2022-08-17T12:23:52.0038225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:52.0038710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:52.0039705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:52.0040243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:52.0173083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:52.0173539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:52.0175859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:52.0176357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:52.1585265Z dist init r=0, world=2 2022-08-17T12:23:52.1700956Z dist init r=1, world=2 2022-08-17T12:23:52.1809780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:52.1810504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:52.1811304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:52.1811863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:52.1915545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:52.1915970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:52.5238798Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:52.5300015Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18246 2022-08-17T12:23:52.5337425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18247 2022-08-17T12:23:53.5017906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:53.5018492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:53.5019869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:53.5020268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:53.5022690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:53.5023224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:53.5026551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:53.5027060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:53.6566192Z dist init r=0, world=2 2022-08-17T12:23:53.6572362Z dist init r=1, world=2 2022-08-17T12:23:53.6775365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:53.6775802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:53.6776439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:53.6776997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:53.6881537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:53.6882114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:53.9368176Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:53.9427428Z test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18274 2022-08-17T12:23:53.9463618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18275 2022-08-17T12:23:54.9110845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:54.9111327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:54.9113360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:54.9113847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:54.9410273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:54.9410642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:54.9413047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:54.9413665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:55.0650580Z dist init r=1, world=2 2022-08-17T12:23:55.1051571Z dist init r=0, world=2 2022-08-17T12:23:55.1261110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:55.1261857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:55.1262694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:55.1263445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:55.1267491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:55.1268094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:55.4494546Z skip: Need at least 2 CUDA devices (1.513s) 2022-08-17T12:23:55.4556427Z test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18302 2022-08-17T12:23:55.4594459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18303 2022-08-17T12:23:56.4234323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:56.4235011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:56.4236126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:56.4236770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:56.4332404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:56.4333125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:56.4335826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:56.4336519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:56.5804937Z dist init r=0, world=2 2022-08-17T12:23:56.5892703Z dist init r=1, world=2 2022-08-17T12:23:56.6002293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:56.6002914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:56.6003546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:56.6004103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:56.6108405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:56.6108952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:56.8624651Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T12:23:56.8680595Z test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18330 2022-08-17T12:23:56.8716045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18331 2022-08-17T12:23:57.8554184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:57.8554593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:57.8556543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:57.8557119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:57.8594554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:57.8594992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:57.8598323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:57.8598700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:58.0106489Z dist init r=0, world=2 2022-08-17T12:23:58.0129707Z dist init r=1, world=2 2022-08-17T12:23:58.0315955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:23:58.0316592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:58.0317240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:58.0317807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:23:58.0422466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:23:58.0422881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:58.3748283Z skip: Need at least 2 CUDA devices (1.512s) 2022-08-17T12:23:58.3803945Z test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18358 2022-08-17T12:23:59.3403329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:23:59.3403822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:23:59.3404744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:23:59.3405189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:23:59.4890314Z dist init r=0, world=1 2022-08-17T12:23:59.4899073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:23:59.4899897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:23:59.4904217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:23:59.7834784Z skip: Need at least 2 CUDA devices (1.408s) 2022-08-17T12:23:59.7889533Z test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18372 2022-08-17T12:24:00.7500955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:00.7523520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:00.7525112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:00.7525657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:00.9029858Z dist init r=0, world=1 2022-08-17T12:24:00.9038109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:00.9038741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:00.9042425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:01.1921281Z skip: Need at least 2 CUDA devices (1.408s) 2022-08-17T12:24:01.1977279Z test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18386 2022-08-17T12:24:02.1626182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:02.1626568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:02.1627854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:02.1628322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:02.3258826Z dist init r=0, world=1 2022-08-17T12:24:02.3266612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:02.3267925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:02.3271494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:02.6008571Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T12:24:02.6064444Z test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18400 2022-08-17T12:24:03.5707200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:03.5707581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:03.5709464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:03.5709847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:03.7234565Z dist init r=0, world=1 2022-08-17T12:24:03.7243119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:03.7243781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:03.7247703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:04.0095499Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T12:24:04.0151017Z test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18414 2022-08-17T12:24:04.9947005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:04.9947500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:04.9948730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:04.9949286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:05.1468474Z dist init r=0, world=1 2022-08-17T12:24:05.1476179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:05.1476818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:05.1481128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:05.4181409Z skip: Need at least 2 CUDA devices (1.408s) 2022-08-17T12:24:05.4237176Z test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18428 2022-08-17T12:24:06.3868655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:06.3869387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:06.3870817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:06.3871188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:06.5415364Z dist init r=0, world=1 2022-08-17T12:24:06.5423751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:06.5424448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:06.5427836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:06.8268800Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T12:24:06.8326875Z test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18442 2022-08-17T12:24:07.8100387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:07.8101041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:07.8102658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:07.8103034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:07.9657896Z dist init r=0, world=1 2022-08-17T12:24:07.9666467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:07.9667377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:07.9671693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:08.2357922Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T12:24:08.2413022Z test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18456 2022-08-17T12:24:09.2041505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:09.2041886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:09.2043429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:09.2043880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:09.3554434Z dist init r=0, world=1 2022-08-17T12:24:09.3563671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:09.3564536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:24:09.3568633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:09.6444612Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T12:24:09.6445163Z 2022-08-17T12:24:09.6445750Z ---------------------------------------------------------------------- 2022-08-17T12:24:09.6446319Z Ran 73 tests in 107.884s 2022-08-17T12:24:09.6446502Z 2022-08-17T12:24:09.6446618Z OK (skipped=73) 2022-08-17T12:24:09.6448738Z 2022-08-17T12:24:09.6448941Z Generating XML reports... 2022-08-17T12:24:09.6557943Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220817122221.xml 2022-08-17T12:24:09.6568476Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220817122221.xml 2022-08-17T12:24:09.9898518Z Running distributed/fsdp/test_fsdp_traversal ... [2022-08-17 12:24:09.989425] 2022-08-17T12:24:09.9899163Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:09.989511] 2022-08-17T12:24:11.1082233Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2022-08-17T12:24:11.1095407Z 2022-08-17T12:24:11.1095490Z Running tests... 2022-08-17T12:24:11.1096099Z ---------------------------------------------------------------------- 2022-08-17T12:24:11.3730453Z test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18481 2022-08-17T12:24:11.3758463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18482 2022-08-17T12:24:12.3760627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:12.3761248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:12.3762414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:12.3763237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:12.3812366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:12.3813004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:12.3814473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:12.3815134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:12.5326167Z dist init r=1, world=2 2022-08-17T12:24:12.5367052Z dist init r=0, world=2 2022-08-17T12:24:12.5568278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:12.5568737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:12.5569619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:24:12.5570193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:24:12.5674699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:12.5675065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:12.8794955Z skip: Need at least 2 CUDA devices (1.770s) 2022-08-17T12:24:12.8795263Z 2022-08-17T12:24:12.8795721Z ---------------------------------------------------------------------- 2022-08-17T12:24:12.8796147Z Ran 1 test in 1.770s 2022-08-17T12:24:12.8796352Z 2022-08-17T12:24:12.8796467Z OK (skipped=1) 2022-08-17T12:24:12.8796642Z 2022-08-17T12:24:12.8796756Z Generating XML reports... 2022-08-17T12:24:12.8830268Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220817122411.xml 2022-08-17T12:24:13.2042642Z Running distributed/fsdp/test_fsdp_uneven ... [2022-08-17 12:24:13.203791] 2022-08-17T12:24:13.2043436Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:13.203881] 2022-08-17T12:24:14.3453218Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-08-17T12:24:14.3464403Z 2022-08-17T12:24:14.3464535Z Running tests... 2022-08-17T12:24:14.3464979Z ---------------------------------------------------------------------- 2022-08-17T12:24:14.3476813Z test_one_iteration (__main__.TestUnevenParamShard) 2022-08-17T12:24:14.6155965Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18520 2022-08-17T12:24:14.6185304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18521 2022-08-17T12:24:14.6213922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18522 2022-08-17T12:24:14.6246648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18523 2022-08-17T12:24:15.6536163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:15.6536806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:15.6538906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:15.6539528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:15.6803346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:15.6803933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:15.6805252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:15.6806179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:15.6953679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:15.6954341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:15.6956205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:15.6956851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:15.6961585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:15.6962226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:15.6965375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:15.6966511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:15.8113376Z dist init r=1, world=4 2022-08-17T12:24:15.8374740Z dist init r=2, world=4 2022-08-17T12:24:15.8548747Z dist init r=0, world=4 2022-08-17T12:24:15.8551153Z dist init r=3, world=4 2022-08-17T12:24:15.8762350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:15.8762856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:15.8863515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:15.8864177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:15.8865281Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:15.8866229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:15.8866863Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:15.8867409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:15.8971829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:15.8972545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:15.8973204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:15.8973820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:16.2291542Z skip: Need at least 2 CUDA devices (1.882s) 2022-08-17T12:24:16.2291777Z 2022-08-17T12:24:16.2292102Z ---------------------------------------------------------------------- 2022-08-17T12:24:16.2292472Z Ran 1 test in 1.883s 2022-08-17T12:24:16.2292593Z 2022-08-17T12:24:16.2292670Z OK (skipped=1) 2022-08-17T12:24:16.2292770Z 2022-08-17T12:24:16.2292860Z Generating XML reports... 2022-08-17T12:24:16.2326306Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220817122414.xml 2022-08-17T12:24:16.5673693Z Running distributed/fsdp/test_shard_utils ... [2022-08-17 12:24:16.566840] 2022-08-17T12:24:16.5674496Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:16.566940] 2022-08-17T12:24:17.9352524Z Running distributed/fsdp/test_utils ... [2022-08-17 12:24:17.934821] 2022-08-17T12:24:17.9353119Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:17.934908] 2022-08-17T12:24:19.0671929Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils 2022-08-17T12:24:19.0682828Z 2022-08-17T12:24:19.0683302Z Running tests... 2022-08-17T12:24:19.0683732Z ---------------------------------------------------------------------- 2022-08-17T12:24:19.3296246Z test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.261s) 2022-08-17T12:24:19.3332402Z test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.004s) 2022-08-17T12:24:19.3353489Z test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s) 2022-08-17T12:24:19.3359962Z test_packed_sequence (__main__.TestUtils) 2022-08-17T12:24:19.3379648Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s) 2022-08-17T12:24:19.3390792Z test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s) 2022-08-17T12:24:19.3391197Z 2022-08-17T12:24:19.3391576Z ---------------------------------------------------------------------- 2022-08-17T12:24:19.3392068Z Ran 5 tests in 0.271s 2022-08-17T12:24:19.3392267Z 2022-08-17T12:24:19.3392396Z OK (skipped=2) 2022-08-17T12:24:19.3392560Z 2022-08-17T12:24:19.3392649Z Generating XML reports... 2022-08-17T12:24:19.3423664Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220817122419.xml 2022-08-17T12:24:19.6361581Z Running distributed/fsdp/test_wrap ... [2022-08-17 12:24:19.635756] 2022-08-17T12:24:19.6362363Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:19.635840] 2022-08-17T12:24:20.7708270Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap 2022-08-17T12:24:20.7724639Z 2022-08-17T12:24:20.7724781Z Running tests... 2022-08-17T12:24:20.7725729Z ---------------------------------------------------------------------- 2022-08-17T12:24:20.7731732Z test_always_wrap (__main__.TestAutoWrap) 2022-08-17T12:24:20.7732434Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s) 2022-08-17T12:24:20.7745099Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7757868Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7763658Z test_auto_wrap_api (__main__.TestAutoWrap) 2022-08-17T12:24:20.7764040Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7771463Z test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap) 2022-08-17T12:24:20.7771870Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7777397Z test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap) 2022-08-17T12:24:20.7777809Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7784663Z test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap) 2022-08-17T12:24:20.7785206Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7792613Z test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap) 2022-08-17T12:24:20.7793071Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7809132Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7825258Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7841174Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7857451Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7873363Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7889192Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7905522Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7921610Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s) 2022-08-17T12:24:20.7934176Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7947094Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7955449Z test_transformer_auto_wrap_policy (__main__.TestAutoWrap) 2022-08-17T12:24:20.7955778Z Tests the ``transformer_auto_wrap_policy``. ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7963928Z test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7969951Z test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7977994Z test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7986486Z test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s) 2022-08-17T12:24:20.7998969Z test_bn_always_wrapped_individually (__main__.TestFSDPWrap) 2022-08-17T12:24:21.0642456Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18607 2022-08-17T12:24:21.0671262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18608 2022-08-17T12:24:21.0701122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18609 2022-08-17T12:24:21.0733027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18610 2022-08-17T12:24:22.0962779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:22.0963464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:22.0964295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:22.0964912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:22.1438535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:22.1439178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:22.1440107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:22.1440719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:22.1511908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:22.1512577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:22.1514866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:22.1515495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:22.1535803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:22.1536473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:22.1539399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:22.1540078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:22.2535507Z dist init r=1, world=4 2022-08-17T12:24:22.3051890Z dist init r=3, world=4 2022-08-17T12:24:22.3115153Z dist init r=0, world=4 2022-08-17T12:24:22.3133857Z dist init r=2, world=4 2022-08-17T12:24:22.3463021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:22.3551356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:22.3653069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:22.3653651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:22.3654652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:22.3655442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:22.3656256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:22.3657124Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:22.3661361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:22.3662124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:22.3662800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:22.3663388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:22.6780271Z skip: Need at least 2 CUDA devices (1.879s) 2022-08-17T12:24:22.6788012Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-08-17T12:24:22.6836387Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18663 2022-08-17T12:24:22.6875684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18664 2022-08-17T12:24:22.6906547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18665 2022-08-17T12:24:22.6936854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18666 2022-08-17T12:24:23.7454589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:23.7455207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:23.7457798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:23.7458408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:23.7490117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:23.7491114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:23.7493969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:23.7494761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:23.7634892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:23.7635572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:23.7638381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:23.7665126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:23.7666061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:23.7666684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:23.7668374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:23.7669021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:23.9069570Z dist init r=2, world=4 2022-08-17T12:24:23.9237751Z dist init r=3, world=4 2022-08-17T12:24:23.9276749Z dist init r=0, world=4 2022-08-17T12:24:23.9286916Z dist init r=1, world=4 2022-08-17T12:24:23.9478277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:23.9546753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:23.9598308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:23.9598927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:23.9599667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:23.9600231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:23.9648667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:23.9681720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:23.9756136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:23.9756920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:23.9757493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:23.9758056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:24.2977447Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:24:24.2985240Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-08-17T12:24:24.3032804Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18719 2022-08-17T12:24:24.3069306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18720 2022-08-17T12:24:24.3098893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18721 2022-08-17T12:24:24.3128164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18722 2022-08-17T12:24:25.3190518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:25.3191171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:25.3192534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:25.3193178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:25.3353224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:25.3353831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:25.3355230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:25.3355841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:25.3404672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:25.3405345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:25.3407555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:25.3408240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:25.3542303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:25.3542922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:25.3544405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:25.3545046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:25.4796697Z dist init r=0, world=4 2022-08-17T12:24:25.4960202Z dist init r=2, world=4 2022-08-17T12:24:25.4994408Z dist init r=3, world=4 2022-08-17T12:24:25.5151432Z dist init r=1, world=4 2022-08-17T12:24:25.5305644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:25.5508725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:25.5509538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:25.5510245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:25.5511285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:25.5511834Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:25.5512564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:25.5513114Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:25.5616314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:25.5616996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:25.5617510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:25.5618081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:25.9168958Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:24:25.9176199Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-08-17T12:24:25.9224779Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18775 2022-08-17T12:24:25.9262573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18776 2022-08-17T12:24:25.9293364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18777 2022-08-17T12:24:25.9324487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18778 2022-08-17T12:24:26.9239034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:26.9239692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:26.9241408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:26.9242029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:26.9313996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:26.9316572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:26.9317481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:26.9318093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:26.9419813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:26.9420514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:26.9421968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:26.9422661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:26.9819115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:26.9819723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:26.9821219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:26.9821834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:27.0863992Z dist init r=2, world=4 2022-08-17T12:24:27.0951254Z dist init r=3, world=4 2022-08-17T12:24:27.1014895Z dist init r=1, world=4 2022-08-17T12:24:27.1552808Z dist init r=0, world=4 2022-08-17T12:24:27.1865832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:27.1966702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:27.2068783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:27.2070803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:27.2072029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:27.2072632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:27.2073187Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:27.2073723Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:27.2076657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:27.2078661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:27.2079194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:27.2079758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:27.5366131Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:24:27.5373844Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-08-17T12:24:27.5423606Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18831 2022-08-17T12:24:27.5463253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18832 2022-08-17T12:24:27.5494443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18833 2022-08-17T12:24:27.5525309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18834 2022-08-17T12:24:28.5329066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:28.5329754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:28.5331414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:28.5332808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:28.5334030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:28.5334664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:28.5338511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:28.5339173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:28.5473932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:28.5474548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:28.5477382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:28.5478058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:28.5901681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:28.5902293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:28.5904282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:28.5904867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:28.6915335Z dist init r=0, world=4 2022-08-17T12:24:28.6923664Z dist init r=2, world=4 2022-08-17T12:24:28.7072514Z dist init r=1, world=4 2022-08-17T12:24:28.7468993Z dist init r=3, world=4 2022-08-17T12:24:28.7638416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:28.7738738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:28.7840221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:28.7840934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:28.7841950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:28.7842814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:28.7843510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:28.7844057Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:28.7948149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:28.7948844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:28.7949708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:28.7950242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:29.1564964Z skip: Need at least 2 CUDA devices (1.620s) 2022-08-17T12:24:29.1636980Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18887 2022-08-17T12:24:29.1672153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18888 2022-08-17T12:24:29.1703606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18889 2022-08-17T12:24:29.1744533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18890 2022-08-17T12:24:30.1588826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:30.1589549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:30.1590892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:30.1591525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:30.1610975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:30.1611582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:30.1614724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:30.1615388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:30.1972513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:30.1973241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:30.1975474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:30.1976018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:30.2236039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:30.2236738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:30.2237929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:30.2238888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:30.3186230Z dist init r=2, world=4 2022-08-17T12:24:30.3214439Z dist init r=3, world=4 2022-08-17T12:24:30.3575455Z dist init r=0, world=4 2022-08-17T12:24:30.3794546Z dist init r=1, world=4 2022-08-17T12:24:30.4028198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:30.4230219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:30.4332614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:30.4333816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:30.4335103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:30.4335789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:30.4336339Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:30.4337066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:30.4341215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:30.4341705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:30.4342156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:30.4342678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:30.7784962Z skip: Need at least 2 CUDA devices (1.622s) 2022-08-17T12:24:30.7859212Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18943 2022-08-17T12:24:30.7896673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18944 2022-08-17T12:24:30.7925509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18945 2022-08-17T12:24:30.7954577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18946 2022-08-17T12:24:31.8381006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:31.8381692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:31.8383225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:31.8383916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:31.8859578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:31.8860252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:31.8861814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:31.8862471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:31.9343879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:31.9344535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:31.9345624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:31.9346254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:31.9478720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:31.9479410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:31.9482631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:31.9483303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:31.9964450Z dist init r=1, world=4 2022-08-17T12:24:32.0435676Z dist init r=0, world=4 2022-08-17T12:24:32.0914475Z dist init r=2, world=4 2022-08-17T12:24:32.1045648Z dist init r=3, world=4 2022-08-17T12:24:32.1224579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:32.1249165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:32.1350212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:32.1350969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:32.1352171Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:32.1352748Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:32.1353285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:32.1428649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:32.1458479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:32.1459230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:32.1459835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:32.1460390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:32.4996778Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:24:32.5070566Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18999 2022-08-17T12:24:32.5107654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19000 2022-08-17T12:24:32.5136523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19001 2022-08-17T12:24:32.5166023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19002 2022-08-17T12:24:33.5017819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:33.5018515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:33.5019908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:33.5020664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:33.5061521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:33.5062197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:33.5065496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:33.5066148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:33.5484015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:33.5484971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:33.5486333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:33.5487022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:33.5805736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:33.5806581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:33.5807510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:33.5808153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:33.6621073Z dist init r=1, world=4 2022-08-17T12:24:33.6678673Z dist init r=2, world=4 2022-08-17T12:24:33.7099250Z dist init r=0, world=4 2022-08-17T12:24:33.7390188Z dist init r=3, world=4 2022-08-17T12:24:33.7511749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:33.7612987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:33.7715681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:33.7716498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:33.7717479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:33.7718048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:33.7718597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:33.7719137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:33.7825336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:33.7826089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:33.7826509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:33.7826856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:34.1207270Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:34.1280756Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19055 2022-08-17T12:24:34.1320035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19056 2022-08-17T12:24:34.1351733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19057 2022-08-17T12:24:34.1383411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19058 2022-08-17T12:24:35.1720958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:35.1721650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:35.1723192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:35.1723884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:35.1877671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:35.1878320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:35.1880228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:35.1880882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:35.2103925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:35.2104540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:35.2106035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:35.2106692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:35.2402583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:35.2403247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:35.2403925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:35.2404447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:35.3312988Z dist init r=2, world=4 2022-08-17T12:24:35.3480659Z dist init r=3, world=4 2022-08-17T12:24:35.3698765Z dist init r=0, world=4 2022-08-17T12:24:35.4001202Z dist init r=1, world=4 2022-08-17T12:24:35.4414342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:35.4414775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:35.4517919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:35.4518640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:35.4519280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:35.4519835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:35.4520393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:35.4520946Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:35.4626884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:35.4627449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:35.4627987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:35.4628629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:35.7423032Z skip: Need at least 2 CUDA devices (1.622s) 2022-08-17T12:24:35.7497629Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19111 2022-08-17T12:24:35.7535114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19112 2022-08-17T12:24:35.7564633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19113 2022-08-17T12:24:35.7595356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19114 2022-08-17T12:24:36.8522441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:36.8523129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:36.8524376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:36.8525312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:36.8966186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:36.8966885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:36.8968407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:36.8969061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:36.9072692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:36.9073383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:36.9075560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:36.9076245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:36.9475110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:36.9476034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:36.9477231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:36.9477878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:37.0095278Z dist init r=0, world=4 2022-08-17T12:24:37.0549914Z dist init r=1, world=4 2022-08-17T12:24:37.0654850Z dist init r=3, world=4 2022-08-17T12:24:37.1033718Z dist init r=2, world=4 2022-08-17T12:24:37.1311241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:37.1311696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:37.1412518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:37.1413122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:37.1414191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:37.1414844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:37.1415393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:37.1415931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:37.1520790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:37.1521422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:37.1522016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:37.1522470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:37.4636176Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:24:37.4709792Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19167 2022-08-17T12:24:37.4747057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19168 2022-08-17T12:24:37.4775974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19169 2022-08-17T12:24:37.4805263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19170 2022-08-17T12:24:38.4716428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:38.4717092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:38.4718717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:38.4719315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:38.4906636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:38.4907241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:38.4908835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:38.4909545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:38.5362296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:38.5362963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:38.5364067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:38.5364737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:38.5608242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:38.5608907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:38.5612149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:38.5612863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:38.6518685Z dist init r=2, world=4 2022-08-17T12:24:38.6551314Z dist init r=1, world=4 2022-08-17T12:24:38.6962763Z dist init r=0, world=4 2022-08-17T12:24:38.7214192Z dist init r=3, world=4 2022-08-17T12:24:38.7322581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:38.7423774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:38.7525181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:38.7525828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:38.7527034Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:38.7527840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:38.7528549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:38.7529096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:38.7571493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:38.7572204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:38.7572734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:38.7573090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:39.0845015Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:39.0918233Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19223 2022-08-17T12:24:39.0956413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19224 2022-08-17T12:24:39.0984849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19225 2022-08-17T12:24:39.1014947Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19226 2022-08-17T12:24:40.1428572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:40.1430523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:40.1431407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:40.1431991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:40.1457872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:40.1458555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:40.1461016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:40.1461913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:40.2244779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:40.2245444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:40.2246865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:40.2247499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:40.2313930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:40.2314597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:40.2316471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:40.2317113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:40.3032506Z dist init r=3, world=4 2022-08-17T12:24:40.3061466Z dist init r=0, world=4 2022-08-17T12:24:40.3824538Z dist init r=1, world=4 2022-08-17T12:24:40.3881703Z dist init r=2, world=4 2022-08-17T12:24:40.4046403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:40.4146689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:40.4192377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:40.4193032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:40.4193793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:40.4194367Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:40.4249634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:40.4250213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:40.4300878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:40.4301470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:40.4302070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:40.4302638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:40.8057198Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:24:40.8132761Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19279 2022-08-17T12:24:40.8171300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19280 2022-08-17T12:24:40.8202724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19281 2022-08-17T12:24:40.8232365Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19282 2022-08-17T12:24:41.9360445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:41.9361146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:41.9362451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:41.9363194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:41.9444806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:41.9445477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:41.9447650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:41.9448316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:41.9596874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:41.9597422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:41.9599340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:41.9599880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:41.9614935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:41.9615499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:41.9618040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:41.9618683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:42.0953668Z dist init r=0, world=4 2022-08-17T12:24:42.1052609Z dist init r=1, world=4 2022-08-17T12:24:42.1205091Z dist init r=3, world=4 2022-08-17T12:24:42.1220370Z dist init r=2, world=4 2022-08-17T12:24:42.1414699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:42.1515567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:42.1617945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:42.1618590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:42.1620533Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:42.1621290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:42.1621991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:42.1622671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:42.1725639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:42.1726406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:42.1726976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:42.1727530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:42.5275618Z skip: Need at least 2 CUDA devices (1.722s) 2022-08-17T12:24:42.5349847Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19335 2022-08-17T12:24:42.5388125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19336 2022-08-17T12:24:42.5419119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19337 2022-08-17T12:24:42.5448551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19338 2022-08-17T12:24:43.5964974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:43.5966133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:43.5967832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:43.5968449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:43.6261946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:43.6262586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:43.6263703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:43.6264370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:43.6545702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:43.6546278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:43.6548182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:43.6548713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:43.6874113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:43.6874773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:43.6876625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:43.6877320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:43.7564299Z dist init r=0, world=4 2022-08-17T12:24:43.7843303Z dist init r=1, world=4 2022-08-17T12:24:43.8113459Z dist init r=3, world=4 2022-08-17T12:24:43.8458658Z dist init r=2, world=4 2022-08-17T12:24:43.8623228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:43.8724600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:43.8770158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:43.8771044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:43.8772350Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:43.8773343Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:43.8827233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:43.8828016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:43.8877443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:43.8878147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:43.8878511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:43.8878872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:44.2490514Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:24:44.2564422Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19391 2022-08-17T12:24:44.2602672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19392 2022-08-17T12:24:44.2631989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19393 2022-08-17T12:24:44.2662481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19394 2022-08-17T12:24:45.2703485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:45.2704105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:45.2705883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:45.2706557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:45.2756236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:45.2756882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:45.2760071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:45.2760758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:45.3092568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:45.3093222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:45.3094520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:45.3095169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:45.3166661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:45.3167339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:45.3169355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:45.3170028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:45.4301271Z dist init r=2, world=4 2022-08-17T12:24:45.4337774Z dist init r=3, world=4 2022-08-17T12:24:45.4697339Z dist init r=0, world=4 2022-08-17T12:24:45.4745938Z dist init r=1, world=4 2022-08-17T12:24:45.4948947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:45.5049585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:45.5114453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:45.5115446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:45.5116522Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:45.5117099Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:45.5152818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:45.5153845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:45.5222102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:45.5222716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:45.5223313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:45.5224322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:45.8703739Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:45.8778709Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19447 2022-08-17T12:24:45.8816707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19448 2022-08-17T12:24:45.8848963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19449 2022-08-17T12:24:45.8878853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19450 2022-08-17T12:24:46.9915968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:46.9916656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:46.9917887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:46.9918688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:47.0100662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:47.0101398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:47.0102831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:47.0103515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:47.0251258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:47.0251873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:47.0254532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:47.0255245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:47.0906116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:47.0906747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:47.0908216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:47.0908859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:47.1522057Z dist init r=0, world=4 2022-08-17T12:24:47.1696014Z dist init r=2, world=4 2022-08-17T12:24:47.1846742Z dist init r=1, world=4 2022-08-17T12:24:47.2446192Z dist init r=3, world=4 2022-08-17T12:24:47.2554716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:47.2561534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:47.2562323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:47.2562853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:47.2563500Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:47.2564051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:47.2564599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:47.2657808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:47.2669399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:47.2670148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:47.2670846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:47.2672880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:47.5920778Z skip: Need at least 2 CUDA devices (1.722s) 2022-08-17T12:24:47.5996228Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19503 2022-08-17T12:24:47.6034215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19504 2022-08-17T12:24:47.6063266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19505 2022-08-17T12:24:47.6094147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19506 2022-08-17T12:24:48.5935857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:48.5936531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:48.5937947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:48.5938579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:48.6117401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:48.6118038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:48.6119760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:48.6120384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:48.6483322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:48.6483999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:48.6484847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:48.6485526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:48.6585824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:48.6587176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:48.6588571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:48.6589306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:48.7546234Z dist init r=0, world=4 2022-08-17T12:24:48.7720703Z dist init r=3, world=4 2022-08-17T12:24:48.8093271Z dist init r=1, world=4 2022-08-17T12:24:48.8178415Z dist init r=2, world=4 2022-08-17T12:24:48.8405674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:48.8406588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:48.8407694Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:48.8408388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:48.8408776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:48.8409313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:48.8410011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:48.8410568Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:48.8513917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:48.8515062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:48.8516142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:48.8516723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:49.2133694Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:49.2207519Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19559 2022-08-17T12:24:49.2244048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19560 2022-08-17T12:24:49.2273188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19561 2022-08-17T12:24:49.2302713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19562 2022-08-17T12:24:50.2384849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:50.2385514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:50.2386680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:50.2387307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:50.2415600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:50.2416286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:50.2419737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:50.2420403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:50.2605722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:50.2606577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:50.2608089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:50.2609013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:50.2948675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:50.2949391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:50.2951110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:50.2951755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:50.3979995Z dist init r=1, world=4 2022-08-17T12:24:50.4011266Z dist init r=0, world=4 2022-08-17T12:24:50.4187991Z dist init r=2, world=4 2022-08-17T12:24:50.4502455Z dist init r=3, world=4 2022-08-17T12:24:50.4693533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:50.4694110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:50.4795432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:50.4796120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:50.4797435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:50.4798456Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:50.4799466Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:50.4800217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:50.4803964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:50.4804419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:50.4805013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:50.4805435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:50.8343256Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:50.8417921Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19615 2022-08-17T12:24:50.8456119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19616 2022-08-17T12:24:50.8486062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19617 2022-08-17T12:24:50.8516111Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19618 2022-08-17T12:24:51.8362596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:51.8363272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:51.8366212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:51.8366895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:51.8586168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:51.8586837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:51.8589672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:51.8590334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:51.8865880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:51.8866573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:51.8868300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:51.8868941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:51.9196933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:51.9197618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:51.9199262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:51.9199912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:51.9948280Z dist init r=3, world=4 2022-08-17T12:24:52.0161081Z dist init r=0, world=4 2022-08-17T12:24:52.0448513Z dist init r=2, world=4 2022-08-17T12:24:52.0761441Z dist init r=1, world=4 2022-08-17T12:24:52.1076779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:52.1077831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:52.1078531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:52.1078980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:52.1079708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:52.1080359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:52.1081122Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:52.1081728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:52.1085395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:52.1086155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:52.1086749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:52.1088540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:52.4555997Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:52.4631980Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19671 2022-08-17T12:24:52.4669838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19672 2022-08-17T12:24:52.4699489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19673 2022-08-17T12:24:52.4728002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19674 2022-08-17T12:24:53.5919924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:53.5920616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:53.5921430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:53.5922095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:53.6002235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:53.6003840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:53.6004941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:53.6005580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:53.6014855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:53.6015498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:53.6019307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:53.6019935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:53.6045519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:53.6046357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:53.6048927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:53.6049545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:53.7521715Z dist init r=0, world=4 2022-08-17T12:24:53.7592022Z dist init r=2, world=4 2022-08-17T12:24:53.7625949Z dist init r=1, world=4 2022-08-17T12:24:53.7641619Z dist init r=3, world=4 2022-08-17T12:24:53.8034457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:53.8135090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:53.8135860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:53.8136558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:53.8137300Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:53.8137855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:53.8138406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:53.8138933Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:53.8243663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:53.8244384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:53.8245236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:53.8246077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:54.1771266Z skip: Need at least 2 CUDA devices (1.721s) 2022-08-17T12:24:54.1844904Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19727 2022-08-17T12:24:54.1883299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19728 2022-08-17T12:24:54.1913364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19729 2022-08-17T12:24:54.1942569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19730 2022-08-17T12:24:55.2634338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:55.2634990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:55.2636322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:55.2636952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:55.2936938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:55.2937563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:55.2940119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:55.2940716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:55.3028941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:55.3029578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:55.3032210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:55.3032853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:55.3115280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:55.3115968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:55.3117696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:55.3118375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:55.4234183Z dist init r=0, world=4 2022-08-17T12:24:55.4579541Z dist init r=1, world=4 2022-08-17T12:24:55.4757676Z dist init r=2, world=4 2022-08-17T12:24:55.4806345Z dist init r=3, world=4 2022-08-17T12:24:55.4915917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:55.5047814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:55.5148449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:55.5149006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:55.5150238Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:55.5150795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:55.5151346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:55.5220915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:55.5257511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:55.5258214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:55.5258902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:55.5314397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:55.7982799Z skip: Need at least 2 CUDA devices (1.621s) 2022-08-17T12:24:55.8039555Z test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19783 2022-08-17T12:24:55.8076878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19784 2022-08-17T12:24:55.8106550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19785 2022-08-17T12:24:55.8136252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19786 2022-08-17T12:24:56.8334636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:56.8335304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:56.8336375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:56.8337024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:56.8359024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:56.8359646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:56.8362652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:56.8363306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:56.8693721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:56.8694311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:56.8695824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:56.8696479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:56.8867667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:56.8868501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:56.8869992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:56.8870692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:56.9938476Z dist init r=2, world=4 2022-08-17T12:24:56.9942444Z dist init r=0, world=4 2022-08-17T12:24:57.0288869Z dist init r=3, world=4 2022-08-17T12:24:57.0446504Z dist init r=1, world=4 2022-08-17T12:24:57.0758995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:57.0759463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:57.0859942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:57.0861128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:57.0862240Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:57.0863112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:57.0863659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:57.0864200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:57.0968261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:57.0969129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:57.0970050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:57.0970417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:57.4177983Z skip: Need at least 2 CUDA devices (1.619s) 2022-08-17T12:24:57.4233755Z test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19839 2022-08-17T12:24:57.4272206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19840 2022-08-17T12:24:57.4301687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19841 2022-08-17T12:24:57.4331211Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19842 2022-08-17T12:24:58.4716405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:58.4717063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:58.4718659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:58.4719288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:58.5100801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:58.5101393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:58.5103470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:58.5104009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:58.5643411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:58.5644174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:58.5645563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:58.5646468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:58.5733130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:24:58.5733779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:24:58.5735317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:24:58.5735999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:24:58.6320476Z dist init r=1, world=4 2022-08-17T12:24:58.6680995Z dist init r=0, world=4 2022-08-17T12:24:58.7246840Z dist init r=3, world=4 2022-08-17T12:24:58.7318987Z dist init r=2, world=4 2022-08-17T12:24:58.7537267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:24:58.7638467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:24:58.7639101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:24:58.7639524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:24:58.7640351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:58.7640930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:58.7641483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:58.7642010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:24:58.7746406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:24:58.7747024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:24:58.7747591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:24:58.7748166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:24:59.1374384Z skip: Need at least 2 CUDA devices (1.719s) 2022-08-17T12:24:59.1374703Z 2022-08-17T12:24:59.1375222Z ---------------------------------------------------------------------- 2022-08-17T12:24:59.1375675Z Ran 46 tests in 38.365s 2022-08-17T12:24:59.1375888Z 2022-08-17T12:24:59.1376024Z OK (skipped=46) 2022-08-17T12:24:59.1376218Z 2022-08-17T12:24:59.1377459Z Generating XML reports... 2022-08-17T12:24:59.1445394Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220817122420.xml 2022-08-17T12:24:59.1469944Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220817122420.xml 2022-08-17T12:24:59.4818129Z Running distributed/nn/jit/test_instantiator ... [2022-08-17 12:24:59.481446] 2022-08-17T12:24:59.4818742Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:24:59.481525] 2022-08-17T12:25:00.6009283Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2022-08-17T12:25:00.6021125Z 2022-08-17T12:25:00.6021461Z Running tests... 2022-08-17T12:25:00.6022136Z ---------------------------------------------------------------------- 2022-08-17T12:25:00.8598128Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.257s) 2022-08-17T12:25:00.8618757Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s) 2022-08-17T12:25:00.8726586Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.011s) 2022-08-17T12:25:00.8726978Z 2022-08-17T12:25:00.8727303Z ---------------------------------------------------------------------- 2022-08-17T12:25:00.8727553Z Ran 3 tests in 0.271s 2022-08-17T12:25:00.8727676Z 2022-08-17T12:25:00.8727740Z OK 2022-08-17T12:25:00.8727840Z 2022-08-17T12:25:00.8727928Z Generating XML reports... 2022-08-17T12:25:00.8755277Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220817122500.xml 2022-08-17T12:25:01.1595188Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-08-17 12:25:01.159153] 2022-08-17T12:25:01.1595860Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:25:01.159233] 2022-08-17T12:25:02.4024659Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-08-17T12:25:02.4040364Z 2022-08-17T12:25:02.4040851Z Running tests... 2022-08-17T12:25:02.4041292Z ---------------------------------------------------------------------- 2022-08-17T12:25:02.4058185Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:02.6587580Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s) 2022-08-17T12:25:02.6600324Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:02.6697067Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19916 2022-08-17T12:25:02.6726916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19917 2022-08-17T12:25:03.8463154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:03.8463860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:03.8735796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:03.8736799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:03.8836725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:03.8837341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:03.9094614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:03.9095256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:03.9996462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:04.0250724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:04.3765338Z skip: CUDA is not available. (1.718s) 2022-08-17T12:25:04.3779258Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3779869Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3791017Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3791616Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3801105Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3801825Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3812269Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3812811Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3823396Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3823968Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3833515Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3834054Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3844132Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3844797Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3855394Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3855921Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3866589Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3867811Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3878271Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3879254Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3890457Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3891470Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3901959Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3902964Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3913807Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3914820Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3925313Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3925992Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3936165Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3936695Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3946889Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.3947651Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:04.3978321Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:04.4027034Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19938 2022-08-17T12:25:04.4066524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19939 2022-08-17T12:25:05.5526260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:05.5526940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:05.5705683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:05.5706092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:05.5896672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:05.5897102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:05.6074826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:05.6075457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:05.7058555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:05.7242574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:06.0099478Z skip: CUDA is not available. (1.615s) 2022-08-17T12:25:06.0132016Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:06.0180892Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19960 2022-08-17T12:25:06.0220269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19961 2022-08-17T12:25:07.2023594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:07.2023996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:07.2255618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:07.2256325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:07.2397786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:07.2398350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:07.2623069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:07.2623732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:07.3562626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:07.3791691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:07.6254622Z skip: CUDA is not available. (1.615s) 2022-08-17T12:25:07.6287318Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:07.6334756Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19982 2022-08-17T12:25:07.6372588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19983 2022-08-17T12:25:08.7907088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:08.7907749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:08.7914085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:08.7914786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:08.8288051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:08.8288833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:08.8298339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:08.8298950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:08.9434277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:08.9448633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:09.2405544Z skip: CUDA is not available. (1.615s) 2022-08-17T12:25:09.2439299Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:09.2488172Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20004 2022-08-17T12:25:09.2526086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20005 2022-08-17T12:25:10.4259830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:10.4260250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:10.4402789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:10.4403161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:10.4633839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:10.4634336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:10.4773721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:10.4774206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:10.5787930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:10.5921320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:10.8563676Z skip: CUDA is not available. (1.616s) 2022-08-17T12:25:10.8597107Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:10.8656199Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20026 2022-08-17T12:25:10.8701064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20027 2022-08-17T12:25:12.0226501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:12.0227233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:12.0313021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:12.0313550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:12.0604073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:12.0604726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:12.0681990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:12.0682708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:12.1765601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:12.1835443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:12.4733448Z skip: CUDA is not available. (1.617s) 2022-08-17T12:25:12.4765752Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:12.4814383Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20048 2022-08-17T12:25:12.4852742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20049 2022-08-17T12:25:13.6447887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:13.6448309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:13.6490406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:13.6490829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:13.6823530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:13.6823948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:13.6871024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:13.6871443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:13.7983683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:13.8034132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:14.0888839Z skip: CUDA is not available. (1.615s) 2022-08-17T12:25:14.0899772Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:14.0947843Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20070 2022-08-17T12:25:14.0987801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20071 2022-08-17T12:25:15.2479644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:15.2480342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:15.2480827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:15.2481254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:15.2842431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:15.2843187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:15.2851853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:15.2852460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:15.4001354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:15.4009602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:15.7023944Z skip: CUDA is not available. (1.613s) 2022-08-17T12:25:15.7045753Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:15.7094291Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20092 2022-08-17T12:25:15.7133824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20093 2022-08-17T12:25:16.8816451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:16.8816883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:16.8844693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:16.8845054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:16.9182706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:16.9183092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:16.9203563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:16.9204093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:17.0343046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:17.0343837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:17.3167771Z skip: CUDA is not available. (1.614s) 2022-08-17T12:25:17.3191570Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:17.3239327Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20114 2022-08-17T12:25:17.3278108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20115 2022-08-17T12:25:18.4768003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:18.4768401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:18.4786369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:18.4786879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:18.5140285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:18.5140754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:18.5157267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:18.5158029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:18.6294160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:18.6295302Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-08-17T12:25:18.6300455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:18.6302293Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-08-17T12:25:18.9312302Z ok (1.614s) 2022-08-17T12:25:18.9321137Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:18.9325834Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-08-17T12:25:18.9340054Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:18.9389105Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20136 2022-08-17T12:25:18.9428053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20137 2022-08-17T12:25:20.1024533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:20.1025244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:20.1050502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:20.1051229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:20.1399140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:20.1399797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:20.1427835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:20.1428268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:20.2567683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:20.2586499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:20.5464226Z skip: CUDA is not available. (1.614s) 2022-08-17T12:25:20.5482522Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:20.5532979Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20158 2022-08-17T12:25:20.5574635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20159 2022-08-17T12:25:21.7074915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:21.7075308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:21.7383702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:21.7384060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:21.7440035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:21.7440478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:21.7753229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:21.7753855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:21.8592020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:21.8905801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:22.1608240Z skip: CUDA is not available. (1.614s) 2022-08-17T12:25:22.1613404Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:22.1662191Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20180 2022-08-17T12:25:22.1700934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20181 2022-08-17T12:25:23.3492172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:23.3492594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:23.3681298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:23.3681823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:23.3860318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:23.3860855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:23.4041470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:23.4041993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:23.5012933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:23.5179292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:23.5392570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:23.5393246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:25:23.5394233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:25:23.5395114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T12:25:23.5474467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9x7k3rtu 2022-08-17T12:25:23.5475492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n5gfuxe 2022-08-17T12:25:23.5476373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9x7k3rtu/_remote_module_non_scriptable.py 2022-08-17T12:25:23.5478134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n5gfuxe/_remote_module_non_scriptable.py 2022-08-17T12:25:23.5637234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:25:23.5637687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:25:23.5918515Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:25:23.5919094Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:25:23.5920404Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:25:23.5921122Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:25:23.8738743Z ok (1.713s) 2022-08-17T12:25:23.8743207Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:23.8744195Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T12:25:23.8750254Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:23.8797411Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20208 2022-08-17T12:25:23.8834612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20209 2022-08-17T12:25:25.0477306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:25.0477779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:25.0832082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:25.0832485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:25.0884349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:25.0884722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:25.1228165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:25.1228735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:25.1986475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:25.2372278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:25.4868546Z skip: Need at least 4 CUDA devices (1.612s) 2022-08-17T12:25:25.4876145Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-08-17T12:25:25.4923491Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20230 2022-08-17T12:25:25.4961312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20231 2022-08-17T12:25:26.6424911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:26.6425299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:26.6435776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:26.6436139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:26.6794252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:26.6794629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:26.6795412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:26.6795856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:26.7950167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:26.7950823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:25:27.0994351Z skip: Need at least 4 CUDA devices (1.612s) 2022-08-17T12:25:27.1012087Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:27.1059016Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20252 2022-08-17T12:25:28.1888098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:28.1888845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:28.2231306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:28.2231956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:28.3367350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:28.3375203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:28.3376156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:28.6095018Z ok (1.510s) 2022-08-17T12:25:28.6105391Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:28.6152974Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20266 2022-08-17T12:25:29.7042202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:29.7042629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:29.7379859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:29.7380245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:29.8514049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:29.8521164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:29.8522070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:30.1184143Z ok (1.509s) 2022-08-17T12:25:30.1192596Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:30.1241191Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20280 2022-08-17T12:25:31.2235007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:31.2235707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:31.2591102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:31.2591751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:31.3734715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:31.3742382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:31.3743301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:31.6272158Z ok (1.509s) 2022-08-17T12:25:31.6295039Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:31.6345902Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20294 2022-08-17T12:25:32.7638959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:32.7639334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:32.7989330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:32.7989710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:32.9123900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:32.9131316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:32.9131908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:33.2377214Z ok (1.610s) 2022-08-17T12:25:33.2387044Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:33.2436052Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20308 2022-08-17T12:25:34.3369174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:34.3369588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:34.3720974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:34.3721359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:34.4860244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:34.4867453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:34.4868713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:34.7467224Z ok (1.509s) 2022-08-17T12:25:34.7477282Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:34.7525097Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20322 2022-08-17T12:25:35.8823634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:35.8824338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:35.9161370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:35.9162038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:36.0299152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:36.0306522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:36.0307365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:36.2554978Z ok (1.509s) 2022-08-17T12:25:36.2563766Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:36.2611683Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20336 2022-08-17T12:25:37.4063496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:37.4064203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:37.4415978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:37.4416623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:37.5570190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:37.5577107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:37.5578084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:37.8643915Z ok (1.609s) 2022-08-17T12:25:37.8652910Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-08-17T12:25:37.8701226Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20350 2022-08-17T12:25:39.0053876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:25:39.0054509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:25:39.0407028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:25:39.0407414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:25:39.1546465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:25:39.1553291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:25:39.1553997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T12:25:39.3732098Z ok (1.509s) 2022-08-17T12:25:39.3732309Z 2022-08-17T12:25:39.3732882Z ---------------------------------------------------------------------- 2022-08-17T12:25:39.3733326Z Ran 42 tests in 36.969s 2022-08-17T12:25:39.3733522Z 2022-08-17T12:25:39.3733644Z OK (skipped=32) 2022-08-17T12:25:39.3733831Z 2022-08-17T12:25:39.3733972Z Generating XML reports... 2022-08-17T12:25:39.3804412Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220817122502.xml 2022-08-17T12:25:39.3814546Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220817122502.xml 2022-08-17T12:25:39.7154677Z Running distributed/pipeline/sync/skip/test_api ... [2022-08-17 12:25:39.714952] 2022-08-17T12:25:39.7155224Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-08-17 12:25:39.715039] 2022-08-17T12:25:40.9962964Z ============================= test session starts ============================== 2022-08-17T12:25:40.9963429Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:41.0018288Z cachedir: .pytest_cache 2022-08-17T12:25:41.0019098Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:41.0019461Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:41.0019718Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:41.0020114Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:41.0137864Z collecting ...  2022-08-17T12:25:41.0138199Z collected 3 items  2022-08-17T12:25:41.0141130Z 2022-08-17T12:25:41.0166257Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-08-17T12:25:41.0178197Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-08-17T12:25:41.0200873Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-08-17T12:25:41.0201922Z 2022-08-17T12:25:41.0202260Z ============================== 3 passed in 0.02s =============================== 2022-08-17T12:25:41.2481240Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-08-17 12:25:41.247746] 2022-08-17T12:25:41.2481763Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-08-17 12:25:41.247836] 2022-08-17T12:25:42.5262278Z ============================= test session starts ============================== 2022-08-17T12:25:42.5263039Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:42.5319220Z cachedir: .pytest_cache 2022-08-17T12:25:42.5320114Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:42.5320705Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:42.5321151Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:42.5322126Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:42.5478541Z collecting ...  2022-08-17T12:25:42.5479116Z collected 13 items  2022-08-17T12:25:42.5482567Z 2022-08-17T12:25:42.5495158Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [ 7%] 2022-08-17T12:25:42.5503376Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%] 2022-08-17T12:25:42.5511832Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%] 2022-08-17T12:25:42.5520118Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%] 2022-08-17T12:25:42.5528570Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%] 2022-08-17T12:25:42.5537007Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%] 2022-08-17T12:25:42.5545205Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%] 2022-08-17T12:25:42.5553580Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%] 2022-08-17T12:25:42.5561822Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%] 2022-08-17T12:25:42.5570431Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%] 2022-08-17T12:25:42.5579850Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%] 2022-08-17T12:25:42.5587982Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%] 2022-08-17T12:25:42.6263274Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED [100%] 2022-08-17T12:25:42.6263743Z 2022-08-17T12:25:42.6264164Z ======================== 1 passed, 12 skipped in 0.10s ========================= 2022-08-17T12:25:42.8792604Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-08-17 12:25:42.878869] 2022-08-17T12:25:42.8793169Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-08-17 12:25:42.878964] 2022-08-17T12:25:44.1579014Z ============================= test session starts ============================== 2022-08-17T12:25:44.1579784Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:44.1635236Z cachedir: .pytest_cache 2022-08-17T12:25:44.1636118Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:44.1636710Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:44.1637144Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:44.1637856Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:44.1764167Z collecting ...  2022-08-17T12:25:44.1764763Z collected 6 items  2022-08-17T12:25:44.1768325Z 2022-08-17T12:25:44.1794213Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-08-17T12:25:44.1808224Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-08-17T12:25:44.1821430Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-08-17T12:25:44.1836228Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-08-17T12:25:44.1849996Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-08-17T12:25:44.1866011Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-08-17T12:25:44.1866570Z 2022-08-17T12:25:44.1866956Z ============================== 6 passed in 0.03s =============================== 2022-08-17T12:25:44.4092733Z Running distributed/pipeline/sync/skip/test_leak ... [2022-08-17 12:25:44.408872] 2022-08-17T12:25:44.4093289Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-08-17 12:25:44.408955] 2022-08-17T12:25:45.7018730Z ============================= test session starts ============================== 2022-08-17T12:25:45.7019204Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:45.7072678Z cachedir: .pytest_cache 2022-08-17T12:25:45.7073509Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:45.7073978Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:45.7074233Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:45.7074612Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:45.7203510Z collecting ...  2022-08-17T12:25:45.7203834Z collected 8 items  2022-08-17T12:25:45.7207487Z 2022-08-17T12:25:45.7897419Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%] 2022-08-17T12:25:45.8065035Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%] 2022-08-17T12:25:45.8305382Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%] 2022-08-17T12:25:45.8484983Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%] 2022-08-17T12:25:45.8704294Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%] 2022-08-17T12:25:45.8906461Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%] 2022-08-17T12:25:45.9051169Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%] 2022-08-17T12:25:45.9267917Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%] 2022-08-17T12:25:45.9268287Z 2022-08-17T12:25:45.9268594Z ============================== 8 passed in 0.23s =============================== 2022-08-17T12:25:46.1593437Z Running distributed/pipeline/sync/skip/test_portal ... [2022-08-17 12:25:46.158995] 2022-08-17T12:25:46.1593969Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-08-17 12:25:46.159078] 2022-08-17T12:25:47.4455338Z ============================= test session starts ============================== 2022-08-17T12:25:47.4455782Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:47.4511199Z cachedir: .pytest_cache 2022-08-17T12:25:47.4512090Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:47.4512561Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:47.4513007Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:47.4513449Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:47.4749350Z collecting ...  2022-08-17T12:25:47.4749725Z collected 10 items  2022-08-17T12:25:47.4752669Z 2022-08-17T12:25:47.4764126Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%] 2022-08-17T12:25:47.4798837Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-08-17T12:25:47.4814443Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-08-17T12:25:47.4826961Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-08-17T12:25:47.4840664Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-08-17T12:25:47.4854162Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-08-17T12:25:47.4867477Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-08-17T12:25:47.4880938Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-08-17T12:25:47.4894474Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-08-17T12:25:47.4910870Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-08-17T12:25:47.4911872Z 2022-08-17T12:25:47.4912324Z ========================= 9 passed, 1 skipped in 0.05s ========================= 2022-08-17T12:25:47.7189594Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-08-17 12:25:47.718562] 2022-08-17T12:25:47.7190119Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-08-17 12:25:47.718650] 2022-08-17T12:25:48.9800618Z ============================= test session starts ============================== 2022-08-17T12:25:48.9801394Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:48.9858905Z cachedir: .pytest_cache 2022-08-17T12:25:48.9859809Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:48.9860370Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:48.9860814Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:48.9861499Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:48.9984407Z collecting ...  2022-08-17T12:25:48.9985001Z collected 7 items  2022-08-17T12:25:48.9988389Z 2022-08-17T12:25:49.0021636Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-08-17T12:25:49.0039188Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-08-17T12:25:49.0055878Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-08-17T12:25:49.0069855Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-08-17T12:25:49.0084520Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-08-17T12:25:49.0097944Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-08-17T12:25:49.0114049Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-08-17T12:25:49.0115041Z 2022-08-17T12:25:49.0115282Z ============================== 7 passed in 0.03s =============================== 2022-08-17T12:25:49.2304725Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-08-17 12:25:49.230016] 2022-08-17T12:25:49.2305460Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-08-17 12:25:49.230102] 2022-08-17T12:25:50.4947732Z ============================= test session starts ============================== 2022-08-17T12:25:50.4948216Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:50.5002655Z cachedir: .pytest_cache 2022-08-17T12:25:50.5003571Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:50.5004049Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:50.5004314Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:50.5004716Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:50.5179252Z collecting ...  2022-08-17T12:25:50.5179611Z collected 6 items  2022-08-17T12:25:50.5182743Z 2022-08-17T12:25:50.5210737Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-08-17T12:25:50.5219612Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%] 2022-08-17T12:25:50.5235601Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-08-17T12:25:50.5248555Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-08-17T12:25:50.5261025Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-08-17T12:25:50.5278255Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-08-17T12:25:50.5279977Z 2022-08-17T12:25:50.5280411Z ========================= 5 passed, 1 skipped in 0.03s ========================= 2022-08-17T12:25:50.7550719Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-08-17 12:25:50.754590] 2022-08-17T12:25:50.7551345Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-08-17 12:25:50.754678] 2022-08-17T12:25:52.0571586Z ============================= test session starts ============================== 2022-08-17T12:25:52.0572045Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:52.0631818Z cachedir: .pytest_cache 2022-08-17T12:25:52.0632660Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:52.0633076Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:52.0633330Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:52.0633955Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:52.0779342Z collecting ...  2022-08-17T12:25:52.0779929Z collected 9 items  2022-08-17T12:25:52.0783016Z 2022-08-17T12:25:52.0812189Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-08-17T12:25:52.0826077Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-08-17T12:25:52.0838698Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-08-17T12:25:52.0852362Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-08-17T12:25:52.0865921Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-08-17T12:25:52.0881244Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-08-17T12:25:52.0893190Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-08-17T12:25:52.0907077Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-08-17T12:25:52.0923861Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-08-17T12:25:52.0925042Z 2022-08-17T12:25:52.0925327Z ============================== 9 passed in 0.04s =============================== 2022-08-17T12:25:52.3250413Z Running distributed/pipeline/sync/test_balance ... [2022-08-17 12:25:52.324656] 2022-08-17T12:25:52.3251173Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-08-17 12:25:52.324743] 2022-08-17T12:25:53.6031507Z ============================= test session starts ============================== 2022-08-17T12:25:53.6032264Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:53.6089086Z cachedir: .pytest_cache 2022-08-17T12:25:53.6090004Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:53.6090611Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:53.6091046Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:53.6091724Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:53.6301872Z collecting ...  2022-08-17T12:25:53.6302457Z collected 15 items  2022-08-17T12:25:53.6305560Z 2022-08-17T12:25:53.6330455Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 6%] 2022-08-17T12:25:53.6342681Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%] 2022-08-17T12:25:53.6355646Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%] 2022-08-17T12:25:53.6367713Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%] 2022-08-17T12:25:53.6376720Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%] 2022-08-17T12:25:54.6414439Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%] 2022-08-17T12:25:54.6423196Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%] 2022-08-17T12:25:54.6431500Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%] 2022-08-17T12:25:54.6439948Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%] 2022-08-17T12:25:54.6465535Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%] 2022-08-17T12:25:55.6497357Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%] 2022-08-17T12:25:56.6540286Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 80%] 2022-08-17T12:25:57.6575798Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%] 2022-08-17T12:25:57.6583719Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%] 2022-08-17T12:25:57.6610814Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-08-17T12:25:57.6611636Z 2022-08-17T12:25:57.6612091Z ======================== 10 passed, 5 skipped in 4.06s ========================= 2022-08-17T12:25:57.8753639Z Running distributed/pipeline/sync/test_bugs ... [2022-08-17 12:25:57.874989] 2022-08-17T12:25:57.8754347Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-08-17 12:25:57.875073] 2022-08-17T12:25:59.1589154Z ============================= test session starts ============================== 2022-08-17T12:25:59.1589633Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:25:59.1641910Z cachedir: .pytest_cache 2022-08-17T12:25:59.1642570Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:25:59.1642919Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:25:59.1643176Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:25:59.1643570Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:25:59.1763960Z collecting ...  2022-08-17T12:25:59.1764288Z collected 4 items  2022-08-17T12:25:59.1767909Z 2022-08-17T12:25:59.2141317Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%] 2022-08-17T12:25:59.2305878Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED [ 50%] 2022-08-17T12:25:59.2315617Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%] 2022-08-17T12:25:59.3192307Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED [100%] 2022-08-17T12:25:59.3192663Z 2022-08-17T12:25:59.3193061Z ========================= 3 passed, 1 skipped in 0.16s ========================= 2022-08-17T12:25:59.5515473Z Running distributed/pipeline/sync/test_checkpoint ... [2022-08-17 12:25:59.551163] 2022-08-17T12:25:59.5516035Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-08-17 12:25:59.551248] 2022-08-17T12:26:00.8218225Z ============================= test session starts ============================== 2022-08-17T12:26:00.8219264Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:00.8282896Z cachedir: .pytest_cache 2022-08-17T12:26:00.8283782Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:00.8284131Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:00.8284381Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:00.8284789Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:00.8420070Z collecting ...  2022-08-17T12:26:00.8420659Z collected 7 items  2022-08-17T12:26:00.8423646Z 2022-08-17T12:26:00.8467638Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%] 2022-08-17T12:26:00.8482649Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%] 2022-08-17T12:26:00.8497972Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%] 2022-08-17T12:26:00.8525825Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%] 2022-08-17T12:26:00.8541601Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%] 2022-08-17T12:26:00.8554870Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%] 2022-08-17T12:26:00.8573999Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-08-17T12:26:00.8574856Z 2022-08-17T12:26:00.8575265Z ============================== 7 passed in 0.04s =============================== 2022-08-17T12:26:01.0920063Z Running distributed/pipeline/sync/test_copy ... [2022-08-17 12:26:01.091607] 2022-08-17T12:26:01.0920802Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-08-17 12:26:01.091687] 2022-08-17T12:26:02.3748264Z ============================= test session starts ============================== 2022-08-17T12:26:02.3748761Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:02.3805043Z cachedir: .pytest_cache 2022-08-17T12:26:02.3806061Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:02.3806512Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:02.3806765Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:02.3807272Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:02.3927868Z collecting ...  2022-08-17T12:26:02.3928231Z collected 5 items  2022-08-17T12:26:02.3930832Z 2022-08-17T12:26:02.3969981Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-08-17T12:26:02.3978592Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED [ 40%] 2022-08-17T12:26:02.3986501Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED [ 60%] 2022-08-17T12:26:02.3996013Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%] 2022-08-17T12:26:02.4013276Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-08-17T12:26:02.4014746Z 2022-08-17T12:26:02.4015046Z ========================= 2 passed, 3 skipped in 0.03s ========================= 2022-08-17T12:26:02.6371726Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-08-17 12:26:02.636782] 2022-08-17T12:26:02.6372300Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-08-17 12:26:02.636868] 2022-08-17T12:26:03.9149930Z ============================= test session starts ============================== 2022-08-17T12:26:03.9150387Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:03.9205031Z cachedir: .pytest_cache 2022-08-17T12:26:03.9206014Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:03.9206626Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:03.9207093Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:03.9207511Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:03.9426332Z collecting ...  2022-08-17T12:26:03.9426896Z collected 11 items  2022-08-17T12:26:03.9429418Z 2022-08-17T12:26:04.0129746Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-08-17T12:26:04.0634503Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-08-17T12:26:04.1029370Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-08-17T12:26:04.1531589Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-08-17T12:26:04.1820645Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-08-17T12:26:04.2046189Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-08-17T12:26:04.2064924Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-08-17T12:26:04.2474162Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-08-17T12:26:04.3989413Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-08-17T12:26:04.5949165Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-08-17T12:26:04.6162840Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-08-17T12:26:04.6163274Z 2022-08-17T12:26:04.6163642Z ============================== 11 passed in 0.70s ============================== 2022-08-17T12:26:04.8465042Z Running distributed/pipeline/sync/test_dependency ... [2022-08-17 12:26:04.846048] 2022-08-17T12:26:04.8465800Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-08-17 12:26:04.846133] 2022-08-17T12:26:06.1307406Z ============================= test session starts ============================== 2022-08-17T12:26:06.1307880Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:06.1364395Z cachedir: .pytest_cache 2022-08-17T12:26:06.1365047Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:06.1365409Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:06.1365647Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:06.1366215Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:06.1550117Z collecting ...  2022-08-17T12:26:06.1550485Z collected 6 items  2022-08-17T12:26:06.1553983Z 2022-08-17T12:26:06.1564712Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED [ 16%] 2022-08-17T12:26:06.1593501Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-08-17T12:26:06.1609192Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-08-17T12:26:06.1625335Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-08-17T12:26:06.1638797Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-08-17T12:26:06.1654903Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-08-17T12:26:06.1656205Z 2022-08-17T12:26:06.1656467Z ========================= 5 passed, 1 skipped in 0.04s ========================= 2022-08-17T12:26:06.3988941Z Running distributed/pipeline/sync/test_inplace ... [2022-08-17 12:26:06.398513] 2022-08-17T12:26:06.3989647Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-08-17 12:26:06.398598] 2022-08-17T12:26:07.6657415Z ============================= test session starts ============================== 2022-08-17T12:26:07.6657855Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:07.6710300Z cachedir: .pytest_cache 2022-08-17T12:26:07.6711146Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:07.6711595Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:07.6711834Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:07.6712224Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:07.6806024Z collecting ...  2022-08-17T12:26:07.6806376Z collected 3 items  2022-08-17T12:26:07.6809760Z 2022-08-17T12:26:07.7353411Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%] 2022-08-17T12:26:07.7499875Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%] 2022-08-17T12:26:07.7664440Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%] 2022-08-17T12:26:07.7665024Z 2022-08-17T12:26:07.7665309Z ========================= 1 passed, 2 xfailed in 0.10s ========================= 2022-08-17T12:26:07.9911050Z Running distributed/pipeline/sync/test_microbatch ... [2022-08-17 12:26:07.990694] 2022-08-17T12:26:07.9911590Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-08-17 12:26:07.990781] 2022-08-17T12:26:09.2543235Z ============================= test session starts ============================== 2022-08-17T12:26:09.2543703Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:09.2602254Z cachedir: .pytest_cache 2022-08-17T12:26:09.2603183Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:09.2603538Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:09.2603782Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:09.2604188Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:09.2838688Z collecting ...  2022-08-17T12:26:09.2839042Z collected 10 items  2022-08-17T12:26:09.2841334Z 2022-08-17T12:26:09.2867022Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-08-17T12:26:09.2880382Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-08-17T12:26:09.2891759Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-08-17T12:26:09.2903552Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-08-17T12:26:09.2915001Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-08-17T12:26:09.2929664Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-08-17T12:26:09.2944590Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-08-17T12:26:09.2956245Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-08-17T12:26:09.2968100Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-08-17T12:26:09.2983160Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-08-17T12:26:09.2984408Z 2022-08-17T12:26:09.2984728Z ============================== 10 passed in 0.04s ============================== 2022-08-17T12:26:09.5172755Z Running distributed/pipeline/sync/test_phony ... [2022-08-17 12:26:09.516851] 2022-08-17T12:26:09.5173269Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-08-17 12:26:09.516933] 2022-08-17T12:26:10.7847562Z ============================= test session starts ============================== 2022-08-17T12:26:10.7848084Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:10.7901226Z cachedir: .pytest_cache 2022-08-17T12:26:10.7901867Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:10.7902261Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:10.7902514Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:10.7902891Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:10.8022410Z collecting ...  2022-08-17T12:26:10.8022980Z collected 4 items  2022-08-17T12:26:10.8025978Z 2022-08-17T12:26:10.8052764Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-08-17T12:26:10.8064724Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-08-17T12:26:10.8075920Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-08-17T12:26:10.8096065Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-08-17T12:26:10.8097387Z 2022-08-17T12:26:10.8097737Z ============================== 4 passed in 0.03s =============================== 2022-08-17T12:26:11.0391718Z Running distributed/pipeline/sync/test_pipe ... [2022-08-17 12:26:11.038764] 2022-08-17T12:26:11.0392261Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-08-17 12:26:11.038850] 2022-08-17T12:26:12.3275859Z ============================= test session starts ============================== 2022-08-17T12:26:12.3276647Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:12.3331213Z cachedir: .pytest_cache 2022-08-17T12:26:12.3332098Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:12.3332712Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:12.3333140Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:12.3352399Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:12.4045717Z collecting ...  2022-08-17T12:26:12.4046560Z collected 56 items  2022-08-17T12:26:12.4049333Z 2022-08-17T12:26:12.4081876Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-08-17T12:26:12.4431804Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED [ 3%] 2022-08-17T12:26:12.4577280Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED [ 5%] 2022-08-17T12:26:12.4745387Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED [ 7%] 2022-08-17T12:26:12.4908640Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED [ 8%] 2022-08-17T12:26:12.5146817Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%] 2022-08-17T12:26:12.5378132Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED [ 12%] 2022-08-17T12:26:12.5626507Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED [ 14%] 2022-08-17T12:26:12.5815194Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%] 2022-08-17T12:26:12.6027398Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%] 2022-08-17T12:26:12.6227995Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED [ 19%] 2022-08-17T12:26:12.6394096Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%] 2022-08-17T12:26:12.6627244Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED [ 23%] 2022-08-17T12:26:12.6866740Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED [ 25%] 2022-08-17T12:26:12.9067804Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%] 2022-08-17T12:26:12.9286972Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED [ 28%] 2022-08-17T12:26:12.9449827Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED [ 30%] 2022-08-17T12:26:12.9626975Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%] 2022-08-17T12:26:12.9783833Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED [ 33%] 2022-08-17T12:26:12.9958804Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED [ 35%] 2022-08-17T12:26:13.0146301Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED [ 37%] 2022-08-17T12:26:13.0346597Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED [ 39%] 2022-08-17T12:26:13.0546681Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%] 2022-08-17T12:26:13.0864658Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%] 2022-08-17T12:26:13.1144729Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%] 2022-08-17T12:26:13.1346166Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%] 2022-08-17T12:26:13.1493808Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%] 2022-08-17T12:26:13.1664081Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%] 2022-08-17T12:26:13.1896524Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%] 2022-08-17T12:26:13.2104344Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%] 2022-08-17T12:26:13.2345496Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%] 2022-08-17T12:26:13.2584408Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED [ 57%] 2022-08-17T12:26:13.2782786Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED [ 58%] 2022-08-17T12:26:13.2943826Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%] 2022-08-17T12:26:13.3364468Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%] 2022-08-17T12:26:13.3588293Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%] 2022-08-17T12:26:13.3826037Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%] 2022-08-17T12:26:13.3998153Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%] 2022-08-17T12:26:13.4258942Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%] 2022-08-17T12:26:13.4439422Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED [ 71%] 2022-08-17T12:26:13.4666510Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED [ 73%] 2022-08-17T12:26:13.4675113Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED [ 75%] 2022-08-17T12:26:13.4946918Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED [ 76%] 2022-08-17T12:26:13.5144285Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED [ 78%] 2022-08-17T12:26:13.5346276Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED [ 80%] 2022-08-17T12:26:13.5528005Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%] 2022-08-17T12:26:13.5749369Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%] 2022-08-17T12:26:13.5757020Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%] 2022-08-17T12:26:13.5764318Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%] 2022-08-17T12:26:13.5946990Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%] 2022-08-17T12:26:13.9125434Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED [ 91%] 2022-08-17T12:26:13.9134257Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%] 2022-08-17T12:26:13.9142198Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%] 2022-08-17T12:26:13.9150181Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%] 2022-08-17T12:26:13.9157940Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%] 2022-08-17T12:26:13.9170613Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%] 2022-08-17T12:26:13.9172137Z 2022-08-17T12:26:13.9172576Z ======================== 48 passed, 8 skipped in 1.59s ========================= 2022-08-17T12:26:14.1527846Z Running distributed/pipeline/sync/test_pipeline ... [2022-08-17 12:26:14.152396] 2022-08-17T12:26:14.1528384Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-08-17 12:26:14.152485] 2022-08-17T12:26:15.4398158Z ============================= test session starts ============================== 2022-08-17T12:26:15.4398638Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:15.4452867Z cachedir: .pytest_cache 2022-08-17T12:26:15.4453526Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:15.4453886Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:15.4454149Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:15.4454550Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:15.4564959Z collecting ...  2022-08-17T12:26:15.4565320Z collected 1 item  2022-08-17T12:26:15.4568584Z 2022-08-17T12:26:15.4596255Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-08-17T12:26:15.4597166Z 2022-08-17T12:26:15.4597394Z ============================== 1 passed in 0.02s =============================== 2022-08-17T12:26:15.6846343Z Running distributed/pipeline/sync/test_stream ... [2022-08-17 12:26:15.684258] 2022-08-17T12:26:15.6846938Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-08-17 12:26:15.684341] 2022-08-17T12:26:16.9758633Z ============================= test session starts ============================== 2022-08-17T12:26:16.9759088Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:16.9813568Z cachedir: .pytest_cache 2022-08-17T12:26:16.9814261Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:16.9814614Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:16.9814867Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:16.9815267Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:17.0042078Z collecting ...  2022-08-17T12:26:17.0042425Z collected 19 items  2022-08-17T12:26:17.0045270Z 2022-08-17T12:26:17.0070357Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-08-17T12:26:17.0079251Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%] 2022-08-17T12:26:17.0091268Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-08-17T12:26:17.0099150Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%] 2022-08-17T12:26:17.0110748Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-08-17T12:26:17.0118386Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%] 2022-08-17T12:26:17.0131851Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-08-17T12:26:17.0139642Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%] 2022-08-17T12:26:17.0151118Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-08-17T12:26:17.0158975Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%] 2022-08-17T12:26:17.0170806Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-08-17T12:26:17.0178438Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%] 2022-08-17T12:26:17.0196155Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-08-17T12:26:17.0203862Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%] 2022-08-17T12:26:17.0211954Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%] 2022-08-17T12:26:17.0219830Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%] 2022-08-17T12:26:17.0232523Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-08-17T12:26:17.0240380Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%] 2022-08-17T12:26:17.0252493Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%] 2022-08-17T12:26:17.0253860Z 2022-08-17T12:26:17.0254277Z ======================== 8 passed, 11 skipped in 0.05s ========================= 2022-08-17T12:26:17.2557853Z Running distributed/pipeline/sync/test_transparency ... [2022-08-17 12:26:17.255319] 2022-08-17T12:26:17.2558398Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-08-17 12:26:17.255404] 2022-08-17T12:26:18.5337410Z ============================= test session starts ============================== 2022-08-17T12:26:18.5337851Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:18.5390751Z cachedir: .pytest_cache 2022-08-17T12:26:18.5391648Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:18.5392018Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:18.5392274Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:18.5392669Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:18.5480511Z collecting ...  2022-08-17T12:26:18.5480851Z collected 1 item  2022-08-17T12:26:18.5484001Z 2022-08-17T12:26:18.6064191Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%] 2022-08-17T12:26:18.6064675Z 2022-08-17T12:26:18.6065132Z ============================== 1 passed in 0.07s =============================== 2022-08-17T12:26:18.8354602Z Running distributed/pipeline/sync/test_worker ... [2022-08-17 12:26:18.835015] 2022-08-17T12:26:18.8355137Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-08-17 12:26:18.835103] 2022-08-17T12:26:20.1033087Z ============================= test session starts ============================== 2022-08-17T12:26:20.1033559Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python 2022-08-17T12:26:20.1087303Z cachedir: .pytest_cache 2022-08-17T12:26:20.1088266Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-08-17T12:26:20.1088876Z torch: 1.13.0a0+gitce6a3c6 2022-08-17T12:26:20.1089140Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-08-17T12:26:20.1089547Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0 2022-08-17T12:26:20.1243089Z collecting ...  2022-08-17T12:26:20.1243534Z collected 6 items  2022-08-17T12:26:20.1246238Z 2022-08-17T12:26:20.1278308Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-08-17T12:26:20.1295231Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-08-17T12:26:20.1309889Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-08-17T12:26:20.1328619Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-08-17T12:26:20.1343510Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-08-17T12:26:20.1363552Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-08-17T12:26:20.1364711Z 2022-08-17T12:26:20.1365087Z ============================== 6 passed in 0.03s =============================== 2022-08-17T12:26:20.3618141Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-08-17 12:26:20.361473] 2022-08-17T12:26:20.3618795Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:26:20.361560] 2022-08-17T12:26:21.4649624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilme4_ey 2022-08-17T12:26:21.4650390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilme4_ey/_remote_module_non_scriptable.py 2022-08-17T12:26:21.7382732Z ]> 2022-08-17T12:26:21.7383645Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) 2022-08-17T12:26:21.7387344Z , <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]> 2022-08-17T12:26:21.7388329Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) 2022-08-17T12:26:21.7389052Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) 2022-08-17T12:26:21.7389679Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) 2022-08-17T12:26:21.7391098Z , <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]> 2022-08-17T12:26:21.7392292Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-08-17T12:26:21.7392954Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) 2022-08-17T12:26:21.7393499Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) 2022-08-17T12:26:21.7393809Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-08-17T12:26:21.7394185Z ]> 2022-08-17T12:26:21.7394545Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) 2022-08-17T12:26:21.7395594Z , <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]> 2022-08-17T12:26:21.7396607Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7396941Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7397254Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7397587Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7397923Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7398237Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7398557Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7398890Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-08-17T12:26:21.7411097Z , <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]> 2022-08-17T12:26:21.7422931Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7423351Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7423760Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7424179Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7424612Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7425041Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7425495Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7425923Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7426351Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7426753Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7427135Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7427497Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7427882Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7428274Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7428735Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7429143Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7429517Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7429875Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7430229Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7430601Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7430966Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7431330Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7431725Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7432110Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7432505Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7432882Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7433256Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7433623Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7433973Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7434333Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7434701Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7435069Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7435417Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7435798Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7436183Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7436546Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7436923Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7437297Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7437666Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7438024Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7438394Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7438809Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7439188Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7439590Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7439984Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7440363Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7440717Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7441108Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7441513Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7441900Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7442283Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7442660Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7443081Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7443474Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7443875Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7444286Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7444681Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7445092Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7445484Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7445988Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7446366Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7446737Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7447111Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7447483Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7447871Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7448249Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7448620Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7448993Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7449404Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7449836Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7450244Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7450640Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7451039Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7451426Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7451795Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7452176Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7452557Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7453017Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7453397Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7453789Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7454183Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7454558Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7454946Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7455334Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7455723Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7456101Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7456493Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-08-17T12:26:21.7457261Z , <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]> 2022-08-17T12:26:21.7457981Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-08-17T12:26:21.7458374Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-08-17T12:26:21.7458785Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-08-17T12:26:22.6867765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:22.6868592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:22.6875337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:22.6876023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:22.8399745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy67wuaax 2022-08-17T12:26:22.8403435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy67wuaax/_remote_module_non_scriptable.py 2022-08-17T12:26:23.0969322Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:23.0980380Z 2022-08-17T12:26:23.0980847Z Running tests... 2022-08-17T12:26:23.0981268Z ---------------------------------------------------------------------- 2022-08-17T12:26:23.1004698Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T12:26:23.1005221Z 2022-08-17T12:26:23.1005629Z ---------------------------------------------------------------------- 2022-08-17T12:26:23.1006272Z Ran 1 test in 0.002s 2022-08-17T12:26:23.1006458Z 2022-08-17T12:26:23.1006581Z OK (skipped=1) 2022-08-17T12:26:23.1006776Z 2022-08-17T12:26:23.1006934Z Generating XML reports... 2022-08-17T12:26:23.1032174Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220817122623.xml 2022-08-17T12:26:24.2648447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:24.2649112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:24.2651711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:24.2652309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:24.4194813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v2zpl9f 2022-08-17T12:26:24.4198302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v2zpl9f/_remote_module_non_scriptable.py 2022-08-17T12:26:24.6803504Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:24.6814887Z 2022-08-17T12:26:24.6815502Z Running tests... 2022-08-17T12:26:24.6816106Z ---------------------------------------------------------------------- 2022-08-17T12:26:24.9430448Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23316 2022-08-17T12:26:24.9457611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23317 2022-08-17T12:26:24.9485450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23318 2022-08-17T12:26:24.9517739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23319 2022-08-17T12:26:26.0377474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:26.0378144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:26.0381731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:26.0382411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:26.0480816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:26.0481464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:26.0484725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:26.0485394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:26.0566459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:26.0567141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:26.0570945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:26.0571604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:26.0973052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:26.0973654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:26.0976798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:26.0977349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:26.1971999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65wd50c8 2022-08-17T12:26:26.1973047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65wd50c8/_remote_module_non_scriptable.py 2022-08-17T12:26:26.2062613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuac7u_4s 2022-08-17T12:26:26.2063729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuac7u_4s/_remote_module_non_scriptable.py 2022-08-17T12:26:26.2152145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q8c2u26 2022-08-17T12:26:26.2153751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q8c2u26/_remote_module_non_scriptable.py 2022-08-17T12:26:26.2564014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu_5xm7p 2022-08-17T12:26:26.2565273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu_5xm7p/_remote_module_non_scriptable.py 2022-08-17T12:26:26.4594186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:26.4665347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:26:26.4773562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:26:26.5200397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:26.8569007Z skip: Need at least 1 CUDA device (2.175s) 2022-08-17T12:26:26.8569341Z 2022-08-17T12:26:26.8569863Z ---------------------------------------------------------------------- 2022-08-17T12:26:26.8570331Z Ran 1 test in 2.175s 2022-08-17T12:26:26.8570553Z 2022-08-17T12:26:26.8571180Z OK (skipped=1) 2022-08-17T12:26:26.8571368Z 2022-08-17T12:26:26.8571512Z Generating XML reports... 2022-08-17T12:26:26.8606528Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122624.xml 2022-08-17T12:26:28.1734885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:28.1735307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:28.1737305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:28.1738018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:28.3226929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcldttnlx 2022-08-17T12:26:28.3230391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcldttnlx/_remote_module_non_scriptable.py 2022-08-17T12:26:28.5844553Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:28.5855328Z 2022-08-17T12:26:28.5855404Z Running tests... 2022-08-17T12:26:28.5856011Z ---------------------------------------------------------------------- 2022-08-17T12:26:28.8545365Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23371 2022-08-17T12:26:28.8572984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23372 2022-08-17T12:26:28.8602657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23373 2022-08-17T12:26:28.8634840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23374 2022-08-17T12:26:29.8849499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:29.8850168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:29.8852858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:29.8853500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:29.9034255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:29.9034949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:29.9038584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:29.9039228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:29.9232037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:29.9232669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:29.9235814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:29.9236456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:29.9266317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:29.9267273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:29.9270553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:29.9271207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:30.0429505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkm_nxq_ 2022-08-17T12:26:30.0430506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkm_nxq_/_remote_module_non_scriptable.py 2022-08-17T12:26:30.0634886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo9i5g8_ 2022-08-17T12:26:30.0635926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo9i5g8_/_remote_module_non_scriptable.py 2022-08-17T12:26:30.0817711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lx2r3ua 2022-08-17T12:26:30.0818499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lx2r3ua/_remote_module_non_scriptable.py 2022-08-17T12:26:30.0840029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvg0lqe5 2022-08-17T12:26:30.0840757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvg0lqe5/_remote_module_non_scriptable.py 2022-08-17T12:26:30.3019750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:30.3246909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:26:30.3404623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:26:30.3556277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:30.6684877Z skip: Need at least 1 CUDA device (2.083s) 2022-08-17T12:26:30.6685204Z 2022-08-17T12:26:30.6685703Z ---------------------------------------------------------------------- 2022-08-17T12:26:30.6686186Z Ran 1 test in 2.083s 2022-08-17T12:26:30.6686310Z 2022-08-17T12:26:30.6686389Z OK (skipped=1) 2022-08-17T12:26:30.6686506Z 2022-08-17T12:26:30.6686595Z Generating XML reports... 2022-08-17T12:26:30.6720698Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122628.xml 2022-08-17T12:26:31.9449541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:31.9449938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:31.9452134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:31.9452540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:32.0943975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod3i3ewx 2022-08-17T12:26:32.0946835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod3i3ewx/_remote_module_non_scriptable.py 2022-08-17T12:26:32.3504549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:32.3515269Z 2022-08-17T12:26:32.3515771Z Running tests... 2022-08-17T12:26:32.3516226Z ---------------------------------------------------------------------- 2022-08-17T12:26:32.6100745Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23426 2022-08-17T12:26:32.6129359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23427 2022-08-17T12:26:32.6160093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23428 2022-08-17T12:26:32.6191054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23429 2022-08-17T12:26:33.6964869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:33.6966043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:33.6967741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:33.6968375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:33.6997933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:33.6998574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:33.7002213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:33.7002904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:33.7009693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:33.7010347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:33.7015031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:33.7015682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:33.7083838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:33.7084504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:33.7087884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:33.7088525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:33.8557541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8i3ylfv 2022-08-17T12:26:33.8558302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8i3ylfv/_remote_module_non_scriptable.py 2022-08-17T12:26:33.8576213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm577gl1b 2022-08-17T12:26:33.8577748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm577gl1b/_remote_module_non_scriptable.py 2022-08-17T12:26:33.8590618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lhe8794 2022-08-17T12:26:33.8592357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lhe8794/_remote_module_non_scriptable.py 2022-08-17T12:26:33.8668086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7p69izh 2022-08-17T12:26:33.8669541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7p69izh/_remote_module_non_scriptable.py 2022-08-17T12:26:34.1180113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:34.1198771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:34.1290390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:26:34.1381462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:26:34.4240756Z skip: Need at least 1 CUDA device (2.072s) 2022-08-17T12:26:34.4241095Z 2022-08-17T12:26:34.4241631Z ---------------------------------------------------------------------- 2022-08-17T12:26:34.4241904Z Ran 1 test in 2.072s 2022-08-17T12:26:34.4242029Z 2022-08-17T12:26:34.4242094Z OK (skipped=1) 2022-08-17T12:26:34.4242208Z 2022-08-17T12:26:34.4242298Z Generating XML reports... 2022-08-17T12:26:34.4277608Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122632.xml 2022-08-17T12:26:35.7081637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:35.7082282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:35.7084569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:35.7084944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:35.8566211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lufmap2 2022-08-17T12:26:35.8569524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lufmap2/_remote_module_non_scriptable.py 2022-08-17T12:26:36.1127505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:36.1138179Z 2022-08-17T12:26:36.1138331Z Running tests... 2022-08-17T12:26:36.1138756Z ---------------------------------------------------------------------- 2022-08-17T12:26:36.3729156Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23481 2022-08-17T12:26:36.3757593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23482 2022-08-17T12:26:37.2955010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:37.2955632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:37.2958211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:37.2958759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:37.3138162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:37.3138658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:37.3142071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:37.3142599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:37.4481969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3oc5e5d 2022-08-17T12:26:37.4483352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3oc5e5d/_remote_module_non_scriptable.py 2022-08-17T12:26:37.4677880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvggb2m2y 2022-08-17T12:26:37.4678575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvggb2m2y/_remote_module_non_scriptable.py 2022-08-17T12:26:37.7051279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:37.7257598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:37.9794720Z skip: Need at least 1 CUDA device (1.865s) 2022-08-17T12:26:37.9795034Z 2022-08-17T12:26:37.9795483Z ---------------------------------------------------------------------- 2022-08-17T12:26:37.9795897Z Ran 1 test in 1.866s 2022-08-17T12:26:37.9796092Z 2022-08-17T12:26:37.9796197Z OK (skipped=1) 2022-08-17T12:26:37.9796363Z 2022-08-17T12:26:37.9796502Z Generating XML reports... 2022-08-17T12:26:37.9833387Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122636.xml 2022-08-17T12:26:39.2759858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:39.2762598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:39.2763429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:39.2764047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:39.4273300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawxjqzbd 2022-08-17T12:26:39.4276515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawxjqzbd/_remote_module_non_scriptable.py 2022-08-17T12:26:39.6843464Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:39.6853785Z 2022-08-17T12:26:39.6853873Z Running tests... 2022-08-17T12:26:39.6854477Z ---------------------------------------------------------------------- 2022-08-17T12:26:39.9499290Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23514 2022-08-17T12:26:39.9528003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23515 2022-08-17T12:26:40.9363195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:40.9363574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:40.9366346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:40.9366739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:40.9760740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:40.9761157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:40.9763988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:40.9764400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:41.0930916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb5fpetq 2022-08-17T12:26:41.0931728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb5fpetq/_remote_module_non_scriptable.py 2022-08-17T12:26:41.1336893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sy9c_gp 2022-08-17T12:26:41.1338492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sy9c_gp/_remote_module_non_scriptable.py 2022-08-17T12:26:41.3594725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:41.3983594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:41.6567330Z skip: Need at least 1 CUDA device (1.971s) 2022-08-17T12:26:41.6567702Z 2022-08-17T12:26:41.6568175Z ---------------------------------------------------------------------- 2022-08-17T12:26:41.6568437Z Ran 1 test in 1.971s 2022-08-17T12:26:41.6568544Z 2022-08-17T12:26:41.6568624Z OK (skipped=1) 2022-08-17T12:26:41.6568739Z 2022-08-17T12:26:41.6568829Z Generating XML reports... 2022-08-17T12:26:41.6604738Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122639.xml 2022-08-17T12:26:42.9844511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:42.9844928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:42.9847193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:42.9847708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:43.1341467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m9rbh1o 2022-08-17T12:26:43.1344818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m9rbh1o/_remote_module_non_scriptable.py 2022-08-17T12:26:43.3895773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:43.3906348Z 2022-08-17T12:26:43.3906498Z Running tests... 2022-08-17T12:26:43.3907097Z ---------------------------------------------------------------------- 2022-08-17T12:26:43.6550724Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23547 2022-08-17T12:26:43.6578660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23548 2022-08-17T12:26:44.6224883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:44.6225470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:44.6227343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:44.6227958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:44.6296514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:44.6297125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:44.6301488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:44.6301896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:44.7778726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhk4cn92 2022-08-17T12:26:44.7779814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhk4cn92/_remote_module_non_scriptable.py 2022-08-17T12:26:44.7840399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzcnt1o_ 2022-08-17T12:26:44.7842231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzcnt1o_/_remote_module_non_scriptable.py 2022-08-17T12:26:45.0367183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:45.0417811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:45.3618214Z skip: Need at least 1 CUDA device (1.971s) 2022-08-17T12:26:45.3618536Z 2022-08-17T12:26:45.3618894Z ---------------------------------------------------------------------- 2022-08-17T12:26:45.3619155Z Ran 1 test in 1.971s 2022-08-17T12:26:45.3619261Z 2022-08-17T12:26:45.3619342Z OK (skipped=1) 2022-08-17T12:26:45.3620750Z 2022-08-17T12:26:45.3620937Z Generating XML reports... 2022-08-17T12:26:45.3655567Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122643.xml 2022-08-17T12:26:46.6693223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:46.6693601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:46.6696190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:46.6696568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:46.8192167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uannxjo 2022-08-17T12:26:46.8195447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uannxjo/_remote_module_non_scriptable.py 2022-08-17T12:26:47.0745047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:47.0756003Z 2022-08-17T12:26:47.0756159Z Running tests... 2022-08-17T12:26:47.0756612Z ---------------------------------------------------------------------- 2022-08-17T12:26:47.3358948Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23580 2022-08-17T12:26:47.3390961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23581 2022-08-17T12:26:48.2921185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:48.2922129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:48.2924795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:48.2925393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:48.2926364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:48.2926947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:48.2929697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:48.2930364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:48.4464131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy52_7yx 2022-08-17T12:26:48.4465070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy52_7yx/_remote_module_non_scriptable.py 2022-08-17T12:26:48.4469128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0e9ros8 2022-08-17T12:26:48.4471978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0e9ros8/_remote_module_non_scriptable.py 2022-08-17T12:26:48.6998754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:48.6999372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:49.0430832Z skip: Need at least 1 CUDA device (1.967s) 2022-08-17T12:26:49.0431133Z 2022-08-17T12:26:49.0431653Z ---------------------------------------------------------------------- 2022-08-17T12:26:49.0431969Z Ran 1 test in 1.967s 2022-08-17T12:26:49.0432090Z 2022-08-17T12:26:49.0432170Z OK (skipped=1) 2022-08-17T12:26:49.0432284Z 2022-08-17T12:26:49.0432376Z Generating XML reports... 2022-08-17T12:26:49.0467343Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122647.xml 2022-08-17T12:26:50.3341995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:50.3342398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:50.3345086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:50.3345445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:50.4868107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_oy8ass 2022-08-17T12:26:50.4871481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_oy8ass/_remote_module_non_scriptable.py 2022-08-17T12:26:50.7439290Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:50.7450373Z 2022-08-17T12:26:50.7450508Z Running tests... 2022-08-17T12:26:50.7450984Z ---------------------------------------------------------------------- 2022-08-17T12:26:51.0092605Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23613 2022-08-17T12:26:51.0120917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23614 2022-08-17T12:26:51.0149760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23615 2022-08-17T12:26:51.0182362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23616 2022-08-17T12:26:52.0618079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:52.0618756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:52.0620509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:52.0621457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:52.0622303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:52.0622891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:52.0626624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:52.0627291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:52.0817343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:52.0818163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:52.0819005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:52.0819641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:52.0820683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:52.0821319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:52.0822434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:52.0823077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:52.2203510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9bb80l6 2022-08-17T12:26:52.2204313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_v9xdtb 2022-08-17T12:26:52.2205089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9bb80l6/_remote_module_non_scriptable.py 2022-08-17T12:26:52.2206035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_v9xdtb/_remote_module_non_scriptable.py 2022-08-17T12:26:52.2394683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnb0fpqj 2022-08-17T12:26:52.2395768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnb0fpqj/_remote_module_non_scriptable.py 2022-08-17T12:26:52.2403428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr77z1ir 2022-08-17T12:26:52.2406211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr77z1ir/_remote_module_non_scriptable.py 2022-08-17T12:26:52.4768770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:26:52.4796775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:52.4994157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:26:52.5002277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:52.8231625Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:26:52.8231912Z 2022-08-17T12:26:52.8232379Z ---------------------------------------------------------------------- 2022-08-17T12:26:52.8232791Z Ran 1 test in 2.078s 2022-08-17T12:26:52.8232980Z 2022-08-17T12:26:52.8233094Z OK (skipped=1) 2022-08-17T12:26:52.8233280Z 2022-08-17T12:26:52.8233427Z Generating XML reports... 2022-08-17T12:26:52.8270060Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220817122650.xml 2022-08-17T12:26:54.0990517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:54.0990906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:54.0993566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:54.0993949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:54.2488200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb8mtlo1 2022-08-17T12:26:54.2491328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb8mtlo1/_remote_module_non_scriptable.py 2022-08-17T12:26:54.5042609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:54.5053575Z 2022-08-17T12:26:54.5053668Z Running tests... 2022-08-17T12:26:54.5054344Z ---------------------------------------------------------------------- 2022-08-17T12:26:54.7724967Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23668 2022-08-17T12:26:54.7752211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23669 2022-08-17T12:26:55.7297706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:55.7298413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:55.7301026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:55.7301618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:55.7364567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:55.7365260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:55.7368279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:55.7368947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:55.8878351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhiy0da7 2022-08-17T12:26:55.8879475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhiy0da7/_remote_module_non_scriptable.py 2022-08-17T12:26:55.8917651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuvgzaco 2022-08-17T12:26:55.8919495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuvgzaco/_remote_module_non_scriptable.py 2022-08-17T12:26:56.1415773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:56.1460387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:26:56.4789872Z skip: Need at least 4 CUDA devices (1.973s) 2022-08-17T12:26:56.4790120Z 2022-08-17T12:26:56.4790634Z ---------------------------------------------------------------------- 2022-08-17T12:26:56.4791116Z Ran 1 test in 1.974s 2022-08-17T12:26:56.4791325Z 2022-08-17T12:26:56.4791431Z OK (skipped=1) 2022-08-17T12:26:56.4791546Z 2022-08-17T12:26:56.4791624Z Generating XML reports... 2022-08-17T12:26:56.4828160Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122654.xml 2022-08-17T12:26:57.7662640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:57.7663028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:57.7664574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:57.7664953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:57.9160967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxp5m8jd 2022-08-17T12:26:57.9164293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxp5m8jd/_remote_module_non_scriptable.py 2022-08-17T12:26:58.1720086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:26:58.1730998Z 2022-08-17T12:26:58.1731102Z Running tests... 2022-08-17T12:26:58.1731769Z ---------------------------------------------------------------------- 2022-08-17T12:26:58.4359670Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23701 2022-08-17T12:26:58.4389405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23702 2022-08-17T12:26:59.4214660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:59.4215354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:59.4218642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:59.4219253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:59.4252363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:26:59.4252994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:26:59.4256146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:26:59.4256800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:26:59.5770182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljek3rl4 2022-08-17T12:26:59.5770876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljek3rl4/_remote_module_non_scriptable.py 2022-08-17T12:26:59.5805658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12c8n_r9 2022-08-17T12:26:59.5807471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12c8n_r9/_remote_module_non_scriptable.py 2022-08-17T12:26:59.8354451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:26:59.8462372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:00.1425659Z skip: Need at least 4 CUDA devices (1.969s) 2022-08-17T12:27:00.1426114Z 2022-08-17T12:27:00.1426688Z ---------------------------------------------------------------------- 2022-08-17T12:27:00.1427115Z Ran 1 test in 1.969s 2022-08-17T12:27:00.1427295Z 2022-08-17T12:27:00.1427441Z OK (skipped=1) 2022-08-17T12:27:00.1427621Z 2022-08-17T12:27:00.1427766Z Generating XML reports... 2022-08-17T12:27:00.1464474Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122658.xml 2022-08-17T12:27:01.4282325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:01.4282721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:01.4285282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:01.4285681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:01.5821270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfr6ooc13 2022-08-17T12:27:01.5824147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfr6ooc13/_remote_module_non_scriptable.py 2022-08-17T12:27:01.8452509Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:01.8463040Z 2022-08-17T12:27:01.8463327Z Running tests... 2022-08-17T12:27:01.8463957Z ---------------------------------------------------------------------- 2022-08-17T12:27:02.1143647Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23734 2022-08-17T12:27:02.1172951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23735 2022-08-17T12:27:03.0612308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:03.0612924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:03.0615432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:03.0615818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:03.0714879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:03.0715228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:03.0718666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:03.0719047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:03.2160934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8xz1oon 2022-08-17T12:27:03.2162719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8xz1oon/_remote_module_non_scriptable.py 2022-08-17T12:27:03.2246999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9scnqz8l 2022-08-17T12:27:03.2248808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9scnqz8l/_remote_module_non_scriptable.py 2022-08-17T12:27:03.4718231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:03.4794990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:03.7210238Z skip: Need at least 4 CUDA devices (1.874s) 2022-08-17T12:27:03.7210521Z 2022-08-17T12:27:03.7211003Z ---------------------------------------------------------------------- 2022-08-17T12:27:03.7211433Z Ran 1 test in 1.875s 2022-08-17T12:27:03.7211602Z 2022-08-17T12:27:03.7211720Z OK (skipped=1) 2022-08-17T12:27:03.7211899Z 2022-08-17T12:27:03.7212068Z Generating XML reports... 2022-08-17T12:27:03.7247802Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122701.xml 2022-08-17T12:27:05.0046108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:05.0046653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:05.0049077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:05.0049469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:05.1542949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eny2kkj 2022-08-17T12:27:05.1546033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eny2kkj/_remote_module_non_scriptable.py 2022-08-17T12:27:05.4122212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:05.4133254Z 2022-08-17T12:27:05.4133351Z Running tests... 2022-08-17T12:27:05.4133927Z ---------------------------------------------------------------------- 2022-08-17T12:27:05.6719774Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23767 2022-08-17T12:27:05.6747364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23768 2022-08-17T12:27:06.6185205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:06.6185605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:06.6188302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:06.6188763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:06.6296345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:06.6296776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:06.6300255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:06.6300737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:06.7718371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl4rbibq 2022-08-17T12:27:06.7720179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl4rbibq/_remote_module_non_scriptable.py 2022-08-17T12:27:06.7858435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzeohuh9q 2022-08-17T12:27:06.7859853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzeohuh9q/_remote_module_non_scriptable.py 2022-08-17T12:27:07.0298836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:07.0468850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:07.3786620Z skip: Need at least 4 CUDA devices (1.965s) 2022-08-17T12:27:07.3786900Z 2022-08-17T12:27:07.3787463Z ---------------------------------------------------------------------- 2022-08-17T12:27:07.3787716Z Ran 1 test in 1.965s 2022-08-17T12:27:07.3787836Z 2022-08-17T12:27:07.3787915Z OK (skipped=1) 2022-08-17T12:27:07.3788028Z 2022-08-17T12:27:07.3788118Z Generating XML reports... 2022-08-17T12:27:07.3824142Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122705.xml 2022-08-17T12:27:08.6808633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:08.6809026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:08.6811439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:08.6811803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:08.8346993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip9odkgj 2022-08-17T12:27:08.8350743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip9odkgj/_remote_module_non_scriptable.py 2022-08-17T12:27:09.0984439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:09.0996242Z 2022-08-17T12:27:09.0996583Z Running tests... 2022-08-17T12:27:09.0997226Z ---------------------------------------------------------------------- 2022-08-17T12:27:09.1002508Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:27:09.1002961Z 2022-08-17T12:27:09.1003337Z ---------------------------------------------------------------------- 2022-08-17T12:27:09.1003777Z Ran 1 test in 0.001s 2022-08-17T12:27:09.1003978Z 2022-08-17T12:27:09.1004093Z OK (skipped=1) 2022-08-17T12:27:09.1004276Z 2022-08-17T12:27:09.1004426Z Generating XML reports... 2022-08-17T12:27:09.1029883Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122709.xml 2022-08-17T12:27:10.2747988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:10.2750499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:10.2751354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:10.2751918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:10.4281542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77h6p2ol 2022-08-17T12:27:10.4284499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77h6p2ol/_remote_module_non_scriptable.py 2022-08-17T12:27:10.6893307Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:10.6904289Z 2022-08-17T12:27:10.6904438Z Running tests... 2022-08-17T12:27:10.6904968Z ---------------------------------------------------------------------- 2022-08-17T12:27:10.6910299Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:27:10.6910861Z 2022-08-17T12:27:10.6911211Z ---------------------------------------------------------------------- 2022-08-17T12:27:10.6911593Z Ran 1 test in 0.001s 2022-08-17T12:27:10.6911719Z 2022-08-17T12:27:10.6911795Z OK (skipped=1) 2022-08-17T12:27:10.6911907Z 2022-08-17T12:27:10.6911997Z Generating XML reports... 2022-08-17T12:27:10.6936982Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122710.xml 2022-08-17T12:27:11.8599564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:11.8600280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:11.8603144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:11.8603768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:12.0115013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl565c2tr 2022-08-17T12:27:12.0118252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl565c2tr/_remote_module_non_scriptable.py 2022-08-17T12:27:12.2692539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:12.2703882Z 2022-08-17T12:27:12.2704267Z Running tests... 2022-08-17T12:27:12.2704913Z ---------------------------------------------------------------------- 2022-08-17T12:27:12.2710341Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:27:12.2711091Z 2022-08-17T12:27:12.2711673Z ---------------------------------------------------------------------- 2022-08-17T12:27:12.2712130Z Ran 1 test in 0.001s 2022-08-17T12:27:12.2712345Z 2022-08-17T12:27:12.2712450Z OK (skipped=1) 2022-08-17T12:27:12.2712571Z 2022-08-17T12:27:12.2712659Z Generating XML reports... 2022-08-17T12:27:12.2736753Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122712.xml 2022-08-17T12:27:13.4262029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:13.4262421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:13.4265133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:13.4265651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:13.5779784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6hjh1s5 2022-08-17T12:27:13.5782998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6hjh1s5/_remote_module_non_scriptable.py 2022-08-17T12:27:13.8401543Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:13.8412783Z 2022-08-17T12:27:13.8412965Z Running tests... 2022-08-17T12:27:13.8413623Z ---------------------------------------------------------------------- 2022-08-17T12:27:13.8433493Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T12:27:13.8434443Z 2022-08-17T12:27:13.8434864Z ---------------------------------------------------------------------- 2022-08-17T12:27:13.8435132Z Ran 1 test in 0.001s 2022-08-17T12:27:13.8435254Z 2022-08-17T12:27:13.8435318Z OK (skipped=1) 2022-08-17T12:27:13.8435436Z 2022-08-17T12:27:13.8435596Z Generating XML reports... 2022-08-17T12:27:13.8445800Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122713.xml 2022-08-17T12:27:15.0113284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:15.0113690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:15.0115845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:15.0116220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:15.1597511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk64p55o 2022-08-17T12:27:15.1601312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk64p55o/_remote_module_non_scriptable.py 2022-08-17T12:27:15.4152250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:15.4163664Z 2022-08-17T12:27:15.4164001Z Running tests... 2022-08-17T12:27:15.4164668Z ---------------------------------------------------------------------- 2022-08-17T12:27:15.6752394Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23840 2022-08-17T12:27:15.6780046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23841 2022-08-17T12:27:15.6808437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23842 2022-08-17T12:27:15.6839912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23843 2022-08-17T12:27:16.7642039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:16.7645677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:16.7646820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:16.7647426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:16.7688897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:16.7689549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:16.7693063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:16.7693713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:16.7805234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:16.7806087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:16.7809123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:16.7810028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:16.7862221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:16.7862898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:16.7866436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:16.7867082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:16.9262353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqny_0xtb 2022-08-17T12:27:16.9263737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqny_0xtb/_remote_module_non_scriptable.py 2022-08-17T12:27:16.9285618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fovkz48 2022-08-17T12:27:16.9288397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fovkz48/_remote_module_non_scriptable.py 2022-08-17T12:27:16.9390934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91gl9_0f 2022-08-17T12:27:16.9391990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91gl9_0f/_remote_module_non_scriptable.py 2022-08-17T12:27:16.9451691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5oxgxiwd 2022-08-17T12:27:16.9454171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5oxgxiwd/_remote_module_non_scriptable.py 2022-08-17T12:27:17.1881319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:17.1890477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:17.1957923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:17.2053319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:17.5891701Z skip: Need at least 1 CUDA device (2.173s) 2022-08-17T12:27:17.5891998Z 2022-08-17T12:27:17.5892532Z ---------------------------------------------------------------------- 2022-08-17T12:27:17.5892864Z Ran 1 test in 2.173s 2022-08-17T12:27:17.5892985Z 2022-08-17T12:27:17.5893061Z OK (skipped=1) 2022-08-17T12:27:17.5893176Z 2022-08-17T12:27:17.5893264Z Generating XML reports... 2022-08-17T12:27:17.5928561Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122715.xml 2022-08-17T12:27:18.8818107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:18.8818517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:18.8820664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:18.8821165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:19.0357169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqrzvliu 2022-08-17T12:27:19.0360617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqrzvliu/_remote_module_non_scriptable.py 2022-08-17T12:27:19.2943187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:19.2953698Z 2022-08-17T12:27:19.2953831Z Running tests... 2022-08-17T12:27:19.2954455Z ---------------------------------------------------------------------- 2022-08-17T12:27:19.5657028Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23895 2022-08-17T12:27:19.5687050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23896 2022-08-17T12:27:19.5716569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23897 2022-08-17T12:27:19.5750304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23898 2022-08-17T12:27:20.5646296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:20.5646953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:20.5650059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:20.5650667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:20.5939862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:20.5940523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:20.5943564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:20.5944191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:20.6069421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:20.6070134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:20.6073024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:20.6073712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:20.6267118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:20.6267672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:20.6269666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:20.6270456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:20.7200975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw58r9fju 2022-08-17T12:27:20.7201820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw58r9fju/_remote_module_non_scriptable.py 2022-08-17T12:27:20.7530139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi23vu_au 2022-08-17T12:27:20.7531018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi23vu_au/_remote_module_non_scriptable.py 2022-08-17T12:27:20.7652382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01rdhfb7 2022-08-17T12:27:20.7653769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01rdhfb7/_remote_module_non_scriptable.py 2022-08-17T12:27:20.7827724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x7c4xnx 2022-08-17T12:27:20.7828625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x7c4xnx/_remote_module_non_scriptable.py 2022-08-17T12:27:20.9804929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:21.0142771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:21.0340076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:21.0422450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:21.3800025Z skip: Need at least 1 CUDA device (2.084s) 2022-08-17T12:27:21.3800297Z 2022-08-17T12:27:21.3800826Z ---------------------------------------------------------------------- 2022-08-17T12:27:21.3801336Z Ran 1 test in 2.085s 2022-08-17T12:27:21.3801550Z 2022-08-17T12:27:21.3801686Z OK (skipped=1) 2022-08-17T12:27:21.3801886Z 2022-08-17T12:27:21.3802028Z Generating XML reports... 2022-08-17T12:27:21.3838179Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122719.xml 2022-08-17T12:27:22.6665157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:22.6665551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:22.6668261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:22.6668786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:22.8156384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp590ckhci 2022-08-17T12:27:22.8159298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp590ckhci/_remote_module_non_scriptable.py 2022-08-17T12:27:23.0717010Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:23.0727482Z 2022-08-17T12:27:23.0727675Z Running tests... 2022-08-17T12:27:23.0728041Z ---------------------------------------------------------------------- 2022-08-17T12:27:23.3319171Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23950 2022-08-17T12:27:23.3348300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23951 2022-08-17T12:27:23.3376557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23952 2022-08-17T12:27:23.3407571Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23953 2022-08-17T12:27:24.4004207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:24.4004874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:24.4008016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:24.4008693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:24.4198683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:24.4199296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:24.4202397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:24.4203061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:24.4208634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:24.4209270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:24.4213821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:24.4214441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:24.4403063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:24.4403751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:24.4407070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:24.4407777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:24.5612734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwryy6nyo 2022-08-17T12:27:24.5614133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwryy6nyo/_remote_module_non_scriptable.py 2022-08-17T12:27:24.5807151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupt8ze0b 2022-08-17T12:27:24.5809572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupt8ze0b/_remote_module_non_scriptable.py 2022-08-17T12:27:24.6026434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u_2f14c 2022-08-17T12:27:24.6027522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u_2f14c/_remote_module_non_scriptable.py 2022-08-17T12:27:24.6028294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgsdfsq_ 2022-08-17T12:27:24.6029897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgsdfsq_/_remote_module_non_scriptable.py 2022-08-17T12:27:24.8262311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:24.8428669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:24.8634364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:24.8728864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:25.1455796Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T12:27:25.1456002Z 2022-08-17T12:27:25.1456967Z ---------------------------------------------------------------------- 2022-08-17T12:27:25.1457378Z Ran 1 test in 2.073s 2022-08-17T12:27:25.1457505Z 2022-08-17T12:27:25.1457573Z OK (skipped=1) 2022-08-17T12:27:25.1457705Z 2022-08-17T12:27:25.1457797Z Generating XML reports... 2022-08-17T12:27:25.1492890Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122723.xml 2022-08-17T12:27:26.4327153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:26.4327549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:26.4330331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:26.4330917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:26.5812584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8hbf7wp 2022-08-17T12:27:26.5815741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8hbf7wp/_remote_module_non_scriptable.py 2022-08-17T12:27:26.8380335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:26.8391267Z 2022-08-17T12:27:26.8391416Z Running tests... 2022-08-17T12:27:26.8392493Z ---------------------------------------------------------------------- 2022-08-17T12:27:27.1007873Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24005 2022-08-17T12:27:27.1036280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24006 2022-08-17T12:27:27.1065709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24007 2022-08-17T12:27:27.1096405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24008 2022-08-17T12:27:28.0888840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:28.0889440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:28.0892096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:28.0892671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:28.1085617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:28.1086496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:28.1089510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:28.1090193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:28.1308886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:28.1309597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:28.1310434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:28.1311072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:28.1311916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:28.1312939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:28.1313830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:28.1314516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:28.2456658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zcxqyjl 2022-08-17T12:27:28.2457470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zcxqyjl/_remote_module_non_scriptable.py 2022-08-17T12:27:28.2658908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobp_mpej 2022-08-17T12:27:28.2659883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobp_mpej/_remote_module_non_scriptable.py 2022-08-17T12:27:28.2894568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs7evfmp 2022-08-17T12:27:28.2895685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs7evfmp/_remote_module_non_scriptable.py 2022-08-17T12:27:28.2904741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnicpurcs 2022-08-17T12:27:28.2907222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnicpurcs/_remote_module_non_scriptable.py 2022-08-17T12:27:28.5057477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:28.5227993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:28.5493221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:28.5586111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:28.9145901Z skip: Need at least 1 CUDA device (2.075s) 2022-08-17T12:27:28.9146243Z 2022-08-17T12:27:28.9146718Z ---------------------------------------------------------------------- 2022-08-17T12:27:28.9147050Z Ran 1 test in 2.075s 2022-08-17T12:27:28.9147169Z 2022-08-17T12:27:28.9147232Z OK (skipped=1) 2022-08-17T12:27:28.9147343Z 2022-08-17T12:27:28.9147435Z Generating XML reports... 2022-08-17T12:27:28.9184110Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122726.xml 2022-08-17T12:27:30.2213578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:30.2213954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:30.2216613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:30.2217117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:30.3715295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplohl_cn1 2022-08-17T12:27:30.3718421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplohl_cn1/_remote_module_non_scriptable.py 2022-08-17T12:27:30.6290428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:30.6301318Z 2022-08-17T12:27:30.6301473Z Running tests... 2022-08-17T12:27:30.6302121Z ---------------------------------------------------------------------- 2022-08-17T12:27:30.8939447Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24060 2022-08-17T12:27:30.8967000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24061 2022-08-17T12:27:30.8995663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24062 2022-08-17T12:27:30.9026042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24063 2022-08-17T12:27:31.9263887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:31.9264945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:31.9266411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:31.9267080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:31.9945903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:31.9946517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:31.9948667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:31.9949267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:32.0381893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:32.0382570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:32.0385579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:32.0386571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:32.0580132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:32.0580802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:32.0583687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:32.0584341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:32.0855133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpky8xd8vs 2022-08-17T12:27:32.0856006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpky8xd8vs/_remote_module_non_scriptable.py 2022-08-17T12:27:32.1540332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiruy6jf9 2022-08-17T12:27:32.1541453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiruy6jf9/_remote_module_non_scriptable.py 2022-08-17T12:27:32.1974052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv41vez2f 2022-08-17T12:27:32.1974890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv41vez2f/_remote_module_non_scriptable.py 2022-08-17T12:27:32.2165488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pgvcbi5 2022-08-17T12:27:32.2166933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pgvcbi5/_remote_module_non_scriptable.py 2022-08-17T12:27:32.3499752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:32.4171175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:32.4620694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:32.4884659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:32.8078430Z skip: Need at least 1 CUDA device (2.177s) 2022-08-17T12:27:32.8078738Z 2022-08-17T12:27:32.8079082Z ---------------------------------------------------------------------- 2022-08-17T12:27:32.8079344Z Ran 1 test in 2.178s 2022-08-17T12:27:32.8079463Z 2022-08-17T12:27:32.8079545Z OK (skipped=1) 2022-08-17T12:27:32.8079718Z 2022-08-17T12:27:32.8079812Z Generating XML reports... 2022-08-17T12:27:32.8115408Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122730.xml 2022-08-17T12:27:34.1093910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:34.1094578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:34.1096975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:34.1097462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:34.2593531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphxyjw87 2022-08-17T12:27:34.2596847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphxyjw87/_remote_module_non_scriptable.py 2022-08-17T12:27:34.5207799Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:34.5218401Z 2022-08-17T12:27:34.5218541Z Running tests... 2022-08-17T12:27:34.5219188Z ---------------------------------------------------------------------- 2022-08-17T12:27:34.7867688Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24115 2022-08-17T12:27:34.7895501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24116 2022-08-17T12:27:34.7922900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24117 2022-08-17T12:27:34.7956548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24118 2022-08-17T12:27:35.7822834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:35.7823508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:35.7826087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:35.7827160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:35.7864446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:35.7865099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:35.7868529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:35.7869172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:35.8123539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:35.8124185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:35.8126695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:35.8127324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:35.8549727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:35.8550405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:35.8553132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:35.8553943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:35.9440438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua86hm5q 2022-08-17T12:27:35.9441224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua86hm5q/_remote_module_non_scriptable.py 2022-08-17T12:27:35.9455631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l8nbhg4 2022-08-17T12:27:35.9457390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l8nbhg4/_remote_module_non_scriptable.py 2022-08-17T12:27:35.9693396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytjtjwlq 2022-08-17T12:27:35.9694514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytjtjwlq/_remote_module_non_scriptable.py 2022-08-17T12:27:36.0108417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy6493v7 2022-08-17T12:27:36.0109486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy6493v7/_remote_module_non_scriptable.py 2022-08-17T12:27:36.2053147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:36.2301500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:36.2304088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:36.2736680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:36.6004852Z skip: Need at least 1 CUDA device (2.078s) 2022-08-17T12:27:36.6005200Z 2022-08-17T12:27:36.6005648Z ---------------------------------------------------------------------- 2022-08-17T12:27:36.6006064Z Ran 1 test in 2.079s 2022-08-17T12:27:36.6006186Z 2022-08-17T12:27:36.6006262Z OK (skipped=1) 2022-08-17T12:27:36.6006376Z 2022-08-17T12:27:36.6006452Z Generating XML reports... 2022-08-17T12:27:36.6041841Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122734.xml 2022-08-17T12:27:37.9189417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:37.9189809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:37.9192263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:37.9192716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:38.0697236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikzrgl0f 2022-08-17T12:27:38.0700604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikzrgl0f/_remote_module_non_scriptable.py 2022-08-17T12:27:38.3251570Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:38.3261837Z 2022-08-17T12:27:38.3262080Z Running tests... 2022-08-17T12:27:38.3262701Z ---------------------------------------------------------------------- 2022-08-17T12:27:38.5855498Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24170 2022-08-17T12:27:38.5882733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24171 2022-08-17T12:27:38.5910531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24172 2022-08-17T12:27:38.5941459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24173 2022-08-17T12:27:39.6573576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:39.6574247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:39.6576695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:39.6577327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:39.6578154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:39.6578757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:39.6582622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:39.6583292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:39.6794076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:39.6795020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:39.6796189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:39.6796844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:39.7105911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:39.7106438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:39.7110194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:39.7110810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:39.8155562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdczzui7 2022-08-17T12:27:39.8156750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdczzui7/_remote_module_non_scriptable.py 2022-08-17T12:27:39.8179119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t9_c6r0 2022-08-17T12:27:39.8181306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t9_c6r0/_remote_module_non_scriptable.py 2022-08-17T12:27:39.8386705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcuxy06w 2022-08-17T12:27:39.8388133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcuxy06w/_remote_module_non_scriptable.py 2022-08-17T12:27:39.8692108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6861028x 2022-08-17T12:27:39.8693287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6861028x/_remote_module_non_scriptable.py 2022-08-17T12:27:40.0755722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:40.0789999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:40.1007745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:40.1514955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:40.4992297Z skip: Need at least 1 CUDA device (2.173s) 2022-08-17T12:27:40.4992641Z 2022-08-17T12:27:40.4993178Z ---------------------------------------------------------------------- 2022-08-17T12:27:40.4993537Z Ran 1 test in 2.173s 2022-08-17T12:27:40.4993657Z 2022-08-17T12:27:40.4993732Z OK (skipped=1) 2022-08-17T12:27:40.4993846Z 2022-08-17T12:27:40.4993924Z Generating XML reports... 2022-08-17T12:27:40.5029490Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122738.xml 2022-08-17T12:27:41.7832502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:41.7832919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:41.7835034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:41.7835402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:41.9366807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnualqsr 2022-08-17T12:27:41.9370077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnualqsr/_remote_module_non_scriptable.py 2022-08-17T12:27:42.1980599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:42.1991751Z 2022-08-17T12:27:42.1991860Z Running tests... 2022-08-17T12:27:42.1992469Z ---------------------------------------------------------------------- 2022-08-17T12:27:42.4685592Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24225 2022-08-17T12:27:42.4715560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24226 2022-08-17T12:27:42.4745770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24227 2022-08-17T12:27:42.4777561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24228 2022-08-17T12:27:43.5218192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:43.5218847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:43.5221452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:43.5222150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:43.5362407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:43.5363070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:43.5366079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:43.5367057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:43.5407592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:43.5408221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:43.5412795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:43.5413489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:43.5643158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:43.5643831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:43.5646319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:43.5646979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:43.6814930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu1c5ft5 2022-08-17T12:27:43.6815748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu1c5ft5/_remote_module_non_scriptable.py 2022-08-17T12:27:43.6949797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_rkxuff 2022-08-17T12:27:43.6950561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_rkxuff/_remote_module_non_scriptable.py 2022-08-17T12:27:43.7006377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fujid3v 2022-08-17T12:27:43.7007581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fujid3v/_remote_module_non_scriptable.py 2022-08-17T12:27:43.7235178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhk3nux9 2022-08-17T12:27:43.7236064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhk3nux9/_remote_module_non_scriptable.py 2022-08-17T12:27:43.9432428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:43.9568266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:43.9724546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:43.9887024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:44.2827061Z skip: Need at least 1 CUDA device (2.083s) 2022-08-17T12:27:44.2827304Z 2022-08-17T12:27:44.2827807Z ---------------------------------------------------------------------- 2022-08-17T12:27:44.2828554Z Ran 1 test in 2.083s 2022-08-17T12:27:44.2828779Z 2022-08-17T12:27:44.2828909Z OK (skipped=1) 2022-08-17T12:27:44.2829031Z 2022-08-17T12:27:44.2829124Z Generating XML reports... 2022-08-17T12:27:44.2867643Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122742.xml 2022-08-17T12:27:45.5655372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:45.5655827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:45.5658285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:45.5658656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:45.7142666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgua_e3vx 2022-08-17T12:27:45.7146193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgua_e3vx/_remote_module_non_scriptable.py 2022-08-17T12:27:45.9687401Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:45.9698037Z 2022-08-17T12:27:45.9698149Z Running tests... 2022-08-17T12:27:45.9699404Z ---------------------------------------------------------------------- 2022-08-17T12:27:46.2274954Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24280 2022-08-17T12:27:46.2304269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24281 2022-08-17T12:27:46.2333230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24282 2022-08-17T12:27:46.2364202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24283 2022-08-17T12:27:47.2569685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:47.2570343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:47.2573329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:47.2574009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:47.2716141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:47.2719457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:47.2720362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:47.2720989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:47.2958560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:47.2959257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:47.2962695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:47.2963382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:47.3128189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:47.3128886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:47.3131889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:47.3132566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:47.4140918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh9aq58i 2022-08-17T12:27:47.4141763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh9aq58i/_remote_module_non_scriptable.py 2022-08-17T12:27:47.4317008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho6qoojn 2022-08-17T12:27:47.4317804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho6qoojn/_remote_module_non_scriptable.py 2022-08-17T12:27:47.4573297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtudxnr0 2022-08-17T12:27:47.4574908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtudxnr0/_remote_module_non_scriptable.py 2022-08-17T12:27:47.4724264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66e9lxfl 2022-08-17T12:27:47.4725273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66e9lxfl/_remote_module_non_scriptable.py 2022-08-17T12:27:47.6773738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:47.6992553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:47.7413913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:47.7656667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:48.0413973Z skip: Need at least 1 CUDA device (2.071s) 2022-08-17T12:27:48.0414517Z 2022-08-17T12:27:48.0415005Z ---------------------------------------------------------------------- 2022-08-17T12:27:48.0415258Z Ran 1 test in 2.072s 2022-08-17T12:27:48.0415378Z 2022-08-17T12:27:48.0415460Z OK (skipped=1) 2022-08-17T12:27:48.0415573Z 2022-08-17T12:27:48.0415663Z Generating XML reports... 2022-08-17T12:27:48.0451441Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122745.xml 2022-08-17T12:27:49.3518649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:49.3519043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:49.3521874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:49.3522251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:49.5044051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpktlac0 2022-08-17T12:27:49.5048175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpktlac0/_remote_module_non_scriptable.py 2022-08-17T12:27:49.7627203Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:49.7638924Z 2022-08-17T12:27:49.7639239Z Running tests... 2022-08-17T12:27:49.7639926Z ---------------------------------------------------------------------- 2022-08-17T12:27:50.0298111Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24335 2022-08-17T12:27:50.0326854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24336 2022-08-17T12:27:50.0356230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24337 2022-08-17T12:27:50.0386481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24338 2022-08-17T12:27:51.0595541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:51.0596179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:51.0596942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:51.0597546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:51.0598341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:51.0599265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:51.0600059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:51.0600682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:51.0656615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:51.0657286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:51.0660196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:51.0660853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:51.0861121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:51.0861785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:51.0864449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:51.0865124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:51.2181904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk9f_jga 2022-08-17T12:27:51.2182998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk9f_jga/_remote_module_non_scriptable.py 2022-08-17T12:27:51.2183677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_glvhr8g 2022-08-17T12:27:51.2186327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_glvhr8g/_remote_module_non_scriptable.py 2022-08-17T12:27:51.2238154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpap89sqmj 2022-08-17T12:27:51.2240511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpap89sqmj/_remote_module_non_scriptable.py 2022-08-17T12:27:51.2445488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0qfzzqj 2022-08-17T12:27:51.2446505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0qfzzqj/_remote_module_non_scriptable.py 2022-08-17T12:27:51.4755414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:51.4780505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:51.4838161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:51.5169207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:51.8437630Z skip: Need at least 1 CUDA device (2.080s) 2022-08-17T12:27:51.8437852Z 2022-08-17T12:27:51.8438249Z ---------------------------------------------------------------------- 2022-08-17T12:27:51.8438501Z Ran 1 test in 2.080s 2022-08-17T12:27:51.8438645Z 2022-08-17T12:27:51.8438722Z OK (skipped=1) 2022-08-17T12:27:51.8438836Z 2022-08-17T12:27:51.8438926Z Generating XML reports... 2022-08-17T12:27:51.8474990Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122749.xml 2022-08-17T12:27:53.1425874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:53.1426349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:53.1428580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:53.1429008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:53.2945937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq46lwv0_ 2022-08-17T12:27:53.2949303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq46lwv0_/_remote_module_non_scriptable.py 2022-08-17T12:27:53.5502504Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:53.5513319Z 2022-08-17T12:27:53.5513450Z Running tests... 2022-08-17T12:27:53.5514075Z ---------------------------------------------------------------------- 2022-08-17T12:27:53.8157626Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24390 2022-08-17T12:27:53.8186024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24391 2022-08-17T12:27:53.8214398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24392 2022-08-17T12:27:53.8244250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24393 2022-08-17T12:27:54.8746610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:54.8747241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:54.8750195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:54.8750884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:54.8929590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:54.8930254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:54.8934010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:54.8934686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:54.9018576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:54.9019199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:54.9022547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:54.9023210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:54.9082054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:54.9082707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:54.9086542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:54.9087212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:55.0350021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzryy9ov 2022-08-17T12:27:55.0350851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzryy9ov/_remote_module_non_scriptable.py 2022-08-17T12:27:55.0526372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz7rq1es 2022-08-17T12:27:55.0527610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz7rq1es/_remote_module_non_scriptable.py 2022-08-17T12:27:55.0602168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfs9w356 2022-08-17T12:27:55.0603102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfs9w356/_remote_module_non_scriptable.py 2022-08-17T12:27:55.0676377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxlbvc2u 2022-08-17T12:27:55.0677835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxlbvc2u/_remote_module_non_scriptable.py 2022-08-17T12:27:55.2962374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:55.3173082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:55.3211391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:55.3289329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:55.6294782Z skip: Need at least 1 CUDA device (2.078s) 2022-08-17T12:27:55.6295072Z 2022-08-17T12:27:55.6295546Z ---------------------------------------------------------------------- 2022-08-17T12:27:55.6295955Z Ran 1 test in 2.078s 2022-08-17T12:27:55.6296155Z 2022-08-17T12:27:55.6296281Z OK (skipped=1) 2022-08-17T12:27:55.6296462Z 2022-08-17T12:27:55.6296598Z Generating XML reports... 2022-08-17T12:27:55.6333234Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122753.xml 2022-08-17T12:27:56.9165687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:56.9166303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:56.9168683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:56.9169081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:57.0651412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_ymw_68 2022-08-17T12:27:57.0654320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_ymw_68/_remote_module_non_scriptable.py 2022-08-17T12:27:57.3201883Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:27:57.3212576Z 2022-08-17T12:27:57.3212768Z Running tests... 2022-08-17T12:27:57.3213222Z ---------------------------------------------------------------------- 2022-08-17T12:27:57.5785569Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24445 2022-08-17T12:27:57.5814248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24446 2022-08-17T12:27:57.5842429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24447 2022-08-17T12:27:57.5873885Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24448 2022-08-17T12:27:58.6476403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:58.6477066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:58.6479850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:58.6480486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:58.6735831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:58.6736490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:58.6739435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:58.6740085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:58.7040076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:58.7040712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:58.7043514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:58.7044164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:58.7096779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:27:58.7097310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:27:58.7100658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:27:58.7101199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:27:58.8051763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrfzwe2n 2022-08-17T12:27:58.8052615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrfzwe2n/_remote_module_non_scriptable.py 2022-08-17T12:27:58.8313220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_pp7n8k 2022-08-17T12:27:58.8314242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_pp7n8k/_remote_module_non_scriptable.py 2022-08-17T12:27:58.8626393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna8eoj34 2022-08-17T12:27:58.8627201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna8eoj34/_remote_module_non_scriptable.py 2022-08-17T12:27:58.8687365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52wr4iqd 2022-08-17T12:27:58.8689590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52wr4iqd/_remote_module_non_scriptable.py 2022-08-17T12:27:59.0667435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:27:59.0932849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:27:59.1433163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:27:59.1562997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:27:59.4925618Z skip: Need at least 1 CUDA device (2.171s) 2022-08-17T12:27:59.4926037Z 2022-08-17T12:27:59.4926783Z ---------------------------------------------------------------------- 2022-08-17T12:27:59.4927037Z Ran 1 test in 2.171s 2022-08-17T12:27:59.4927163Z 2022-08-17T12:27:59.4927242Z OK (skipped=1) 2022-08-17T12:27:59.4927398Z 2022-08-17T12:27:59.4927533Z Generating XML reports... 2022-08-17T12:27:59.4963873Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122757.xml 2022-08-17T12:28:00.7851864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:00.7852249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:00.7855276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:00.7855651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:00.9409710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wo_yswv 2022-08-17T12:28:00.9410745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wo_yswv/_remote_module_non_scriptable.py 2022-08-17T12:28:01.2034101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:01.2043885Z 2022-08-17T12:28:01.2044034Z Running tests... 2022-08-17T12:28:01.2044628Z ---------------------------------------------------------------------- 2022-08-17T12:28:01.4730713Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24500 2022-08-17T12:28:01.4760541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24501 2022-08-17T12:28:01.4790498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24502 2022-08-17T12:28:01.4821420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24503 2022-08-17T12:28:02.5038623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:02.5039270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:02.5042407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:02.5043034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:02.5052580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:02.5053222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:02.5057188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:02.5057821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:02.5238181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:02.5238822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:02.5241979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:02.5242656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:02.5345504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:02.5346171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:02.5350051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:02.5350688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:02.6638427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0va42s_o 2022-08-17T12:28:02.6639260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0va42s_o/_remote_module_non_scriptable.py 2022-08-17T12:28:02.6658512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5n1de8bu 2022-08-17T12:28:02.6660022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5n1de8bu/_remote_module_non_scriptable.py 2022-08-17T12:28:02.6847636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckjxgex2 2022-08-17T12:28:02.6848429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckjxgex2/_remote_module_non_scriptable.py 2022-08-17T12:28:02.6920533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf48mtn1 2022-08-17T12:28:02.6921633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf48mtn1/_remote_module_non_scriptable.py 2022-08-17T12:28:02.9247006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:02.9280170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:02.9519490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:02.9556648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:03.2872605Z skip: Need at least 1 CUDA device (2.082s) 2022-08-17T12:28:03.2872903Z 2022-08-17T12:28:03.2873451Z ---------------------------------------------------------------------- 2022-08-17T12:28:03.2873731Z Ran 1 test in 2.083s 2022-08-17T12:28:03.2873851Z 2022-08-17T12:28:03.2873915Z OK (skipped=1) 2022-08-17T12:28:03.2874032Z 2022-08-17T12:28:03.2874124Z Generating XML reports... 2022-08-17T12:28:03.2909367Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122801.xml 2022-08-17T12:28:04.5849981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:04.5850404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:04.5852990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:04.5853629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:04.7352103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m6ydqq0 2022-08-17T12:28:04.7355485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m6ydqq0/_remote_module_non_scriptable.py 2022-08-17T12:28:04.9922992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:04.9934228Z 2022-08-17T12:28:04.9934636Z Running tests... 2022-08-17T12:28:04.9935038Z ---------------------------------------------------------------------- 2022-08-17T12:28:05.2582766Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24555 2022-08-17T12:28:05.2611519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24556 2022-08-17T12:28:05.2639006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24557 2022-08-17T12:28:05.2668814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24558 2022-08-17T12:28:06.3179312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:06.3180192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:06.3181019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:06.3181666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:06.3255545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:06.3256170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:06.3259999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:06.3260684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:06.3498382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:06.3499074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:06.3501445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:06.3502762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:06.3536159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:06.3536775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:06.3541681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:06.3542454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:06.4801156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeplw52f_ 2022-08-17T12:28:06.4802212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeplw52f_/_remote_module_non_scriptable.py 2022-08-17T12:28:06.4861900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69udh6u5 2022-08-17T12:28:06.4863417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69udh6u5/_remote_module_non_scriptable.py 2022-08-17T12:28:06.5089746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw6b6jmr 2022-08-17T12:28:06.5090571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw6b6jmr/_remote_module_non_scriptable.py 2022-08-17T12:28:06.5126056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x105ssz 2022-08-17T12:28:06.5127694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x105ssz/_remote_module_non_scriptable.py 2022-08-17T12:28:06.7430861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:06.7477817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:06.7686346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:06.7724168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:07.0718473Z skip: Need at least 1 CUDA device (2.078s) 2022-08-17T12:28:07.0718766Z 2022-08-17T12:28:07.0719143Z ---------------------------------------------------------------------- 2022-08-17T12:28:07.0719491Z Ran 1 test in 2.078s 2022-08-17T12:28:07.0719661Z 2022-08-17T12:28:07.0719738Z OK (skipped=1) 2022-08-17T12:28:07.0719851Z 2022-08-17T12:28:07.0719951Z Generating XML reports... 2022-08-17T12:28:07.0756075Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122804.xml 2022-08-17T12:28:08.3845395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:08.3846019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:08.3848147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:08.3848536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:08.5342252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf4lx5vd 2022-08-17T12:28:08.5345714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf4lx5vd/_remote_module_non_scriptable.py 2022-08-17T12:28:08.7907418Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:08.7918601Z 2022-08-17T12:28:08.7918746Z Running tests... 2022-08-17T12:28:08.7919170Z ---------------------------------------------------------------------- 2022-08-17T12:28:09.0558074Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24610 2022-08-17T12:28:09.0589080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24611 2022-08-17T12:28:09.0617665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24612 2022-08-17T12:28:09.0649371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24613 2022-08-17T12:28:10.0648678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:10.0649343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:10.0653143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:10.0653844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:10.0981366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:10.0982002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:10.0982749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:10.0983343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:10.0984142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:10.0984698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:10.0985487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:10.0986432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:10.1054106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:10.1054822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:10.1058013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:10.1058717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:10.2208057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpvne9ak 2022-08-17T12:28:10.2210451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpvne9ak/_remote_module_non_scriptable.py 2022-08-17T12:28:10.2581743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01u6nerk 2022-08-17T12:28:10.2582648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01u6nerk/_remote_module_non_scriptable.py 2022-08-17T12:28:10.2585152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcc82cno 2022-08-17T12:28:10.2587803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcc82cno/_remote_module_non_scriptable.py 2022-08-17T12:28:10.2646737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8jhm3vf 2022-08-17T12:28:10.2648329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8jhm3vf/_remote_module_non_scriptable.py 2022-08-17T12:28:10.4816548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:10.5207166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:10.5440072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:10.5462943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:10.8699497Z skip: Need at least 1 CUDA device (2.078s) 2022-08-17T12:28:10.8699761Z 2022-08-17T12:28:10.8700116Z ---------------------------------------------------------------------- 2022-08-17T12:28:10.8700384Z Ran 1 test in 2.078s 2022-08-17T12:28:10.8700602Z 2022-08-17T12:28:10.8700680Z OK (skipped=1) 2022-08-17T12:28:10.8700795Z 2022-08-17T12:28:10.8700882Z Generating XML reports... 2022-08-17T12:28:10.8736426Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122808.xml 2022-08-17T12:28:12.1583714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:12.1584125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:12.1586504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:12.1586911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:12.3077426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfzxu71n 2022-08-17T12:28:12.3080452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfzxu71n/_remote_module_non_scriptable.py 2022-08-17T12:28:12.5639846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:12.5650904Z 2022-08-17T12:28:12.5651037Z Running tests... 2022-08-17T12:28:12.5651673Z ---------------------------------------------------------------------- 2022-08-17T12:28:12.8273065Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24665 2022-08-17T12:28:12.8301029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24666 2022-08-17T12:28:12.8330228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24667 2022-08-17T12:28:12.8362443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24668 2022-08-17T12:28:13.8176849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:13.8177451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:13.8180372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:13.8180966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:13.8606539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:13.8607243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:13.8610248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:13.8610968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:13.8741949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:13.8742832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:13.8745237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:13.8745824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:13.9054284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:13.9054907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:13.9057954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:13.9058620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:13.9779029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp837mhjis 2022-08-17T12:28:13.9779864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp837mhjis/_remote_module_non_scriptable.py 2022-08-17T12:28:14.0211185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1lkr55c 2022-08-17T12:28:14.0211954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1lkr55c/_remote_module_non_scriptable.py 2022-08-17T12:28:14.0337900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9dm9y6v 2022-08-17T12:28:14.0339198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9dm9y6v/_remote_module_non_scriptable.py 2022-08-17T12:28:14.0660714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3_ua3tt 2022-08-17T12:28:14.0661890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3_ua3tt/_remote_module_non_scriptable.py 2022-08-17T12:28:14.2432780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:14.2859135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:14.3046220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:14.3449379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:14.6412277Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T12:28:14.6412525Z 2022-08-17T12:28:14.6412881Z ---------------------------------------------------------------------- 2022-08-17T12:28:14.6413265Z Ran 1 test in 2.076s 2022-08-17T12:28:14.6413388Z 2022-08-17T12:28:14.6413473Z OK (skipped=1) 2022-08-17T12:28:14.6413591Z 2022-08-17T12:28:14.6414164Z Generating XML reports... 2022-08-17T12:28:14.6451511Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122812.xml 2022-08-17T12:28:15.9355701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:15.9356126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:15.9358823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:15.9359203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:16.0874674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50qfee95 2022-08-17T12:28:16.0877970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50qfee95/_remote_module_non_scriptable.py 2022-08-17T12:28:16.3430240Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:16.3440550Z 2022-08-17T12:28:16.3440716Z Running tests... 2022-08-17T12:28:16.3441134Z ---------------------------------------------------------------------- 2022-08-17T12:28:16.5986425Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s) 2022-08-17T12:28:16.5986978Z 2022-08-17T12:28:16.5987213Z ---------------------------------------------------------------------- 2022-08-17T12:28:16.5987470Z Ran 1 test in 0.254s 2022-08-17T12:28:16.5987592Z 2022-08-17T12:28:16.5987668Z OK (skipped=1) 2022-08-17T12:28:16.5987780Z 2022-08-17T12:28:16.5987855Z Generating XML reports... 2022-08-17T12:28:16.6013332Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122816.xml 2022-08-17T12:28:17.8508589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:17.8508980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:17.8511325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:17.8511687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:18.0019188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bvq40yj 2022-08-17T12:28:18.0021764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bvq40yj/_remote_module_non_scriptable.py 2022-08-17T12:28:18.2579156Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:18.2589904Z 2022-08-17T12:28:18.2590016Z Running tests... 2022-08-17T12:28:18.2590683Z ---------------------------------------------------------------------- 2022-08-17T12:28:18.5203539Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24730 2022-08-17T12:28:18.5231582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24731 2022-08-17T12:28:18.5260577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24732 2022-08-17T12:28:18.5291330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24733 2022-08-17T12:28:19.5536006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:19.5536709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:19.5539195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:19.5540359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:19.6026600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:19.6027298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:19.6028627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:19.6029602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:19.6256328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:19.6257006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:19.6259441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:19.6260116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:19.6342848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:19.6343522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:19.6346661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:19.6347366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:19.7145751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfffsxj77 2022-08-17T12:28:19.7146590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfffsxj77/_remote_module_non_scriptable.py 2022-08-17T12:28:19.7622374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3q1wfva 2022-08-17T12:28:19.7623209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3q1wfva/_remote_module_non_scriptable.py 2022-08-17T12:28:19.7862912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tsyzjk_ 2022-08-17T12:28:19.7863629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tsyzjk_/_remote_module_non_scriptable.py 2022-08-17T12:28:19.8153096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bonwa9k 2022-08-17T12:28:19.8153931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bonwa9k/_remote_module_non_scriptable.py 2022-08-17T12:28:19.9864547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:20.0287046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:20.0508352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:20.0809508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:20.4343399Z skip: Need at least 2 CUDA devices (2.175s) 2022-08-17T12:28:20.4343764Z 2022-08-17T12:28:20.4344197Z ---------------------------------------------------------------------- 2022-08-17T12:28:20.4344442Z Ran 1 test in 2.175s 2022-08-17T12:28:20.4344564Z 2022-08-17T12:28:20.4344642Z OK (skipped=1) 2022-08-17T12:28:20.4344757Z 2022-08-17T12:28:20.4344853Z Generating XML reports... 2022-08-17T12:28:20.4379962Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122818.xml 2022-08-17T12:28:21.7133634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:21.7134055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:21.7136447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:21.7136952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:21.8716828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshx6k_qp 2022-08-17T12:28:21.8720012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshx6k_qp/_remote_module_non_scriptable.py 2022-08-17T12:28:22.1330818Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:22.1341371Z 2022-08-17T12:28:22.1341515Z Running tests... 2022-08-17T12:28:22.1341932Z ---------------------------------------------------------------------- 2022-08-17T12:28:22.3955165Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24785 2022-08-17T12:28:22.3982180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24786 2022-08-17T12:28:22.4010756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24787 2022-08-17T12:28:22.4041382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24788 2022-08-17T12:28:23.4039990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:23.4040672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:23.4052456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:23.4053148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:23.4245448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:23.4246383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:23.4247175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:23.4247793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:23.4248619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:23.4249270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:23.4251368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:23.4252014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:23.4417489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:23.4418169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:23.4421421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:23.4422305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:23.5644577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2hdhf41 2022-08-17T12:28:23.5645307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2hdhf41/_remote_module_non_scriptable.py 2022-08-17T12:28:23.5836087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hp399ig 2022-08-17T12:28:23.5837123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hp399ig/_remote_module_non_scriptable.py 2022-08-17T12:28:23.5849933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsazu5o4k 2022-08-17T12:28:23.5851971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsazu5o4k/_remote_module_non_scriptable.py 2022-08-17T12:28:23.6009260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb79izx2 2022-08-17T12:28:23.6010078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb79izx2/_remote_module_non_scriptable.py 2022-08-17T12:28:23.8246893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:23.8431101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:23.8436619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:23.8721565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:24.2091301Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T12:28:24.2091525Z 2022-08-17T12:28:24.2092002Z ---------------------------------------------------------------------- 2022-08-17T12:28:24.2092484Z Ran 1 test in 2.075s 2022-08-17T12:28:24.2092699Z 2022-08-17T12:28:24.2092792Z OK (skipped=1) 2022-08-17T12:28:24.2092910Z 2022-08-17T12:28:24.2093002Z Generating XML reports... 2022-08-17T12:28:24.2129295Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122822.xml 2022-08-17T12:28:25.5133342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:25.5133800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:25.5136878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:25.5137266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:25.6648971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzm8uyo7 2022-08-17T12:28:25.6652683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzm8uyo7/_remote_module_non_scriptable.py 2022-08-17T12:28:25.9229222Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:25.9240367Z 2022-08-17T12:28:25.9240510Z Running tests... 2022-08-17T12:28:25.9241091Z ---------------------------------------------------------------------- 2022-08-17T12:28:26.1970962Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24840 2022-08-17T12:28:26.1999196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24841 2022-08-17T12:28:26.2027420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24842 2022-08-17T12:28:26.2058237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24843 2022-08-17T12:28:27.2987118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:27.2987816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:27.2990606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:27.2991233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:27.3070880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:27.3071492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:27.3075003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:27.3075695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:27.3130146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:27.3133625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:27.3134497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:27.3135144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:27.3358121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:27.3358710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:27.3362324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:27.3363008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:27.4589961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dhs0xub 2022-08-17T12:28:27.4590789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dhs0xub/_remote_module_non_scriptable.py 2022-08-17T12:28:27.4658185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lietp9u 2022-08-17T12:28:27.4660687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lietp9u/_remote_module_non_scriptable.py 2022-08-17T12:28:27.4711682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61i2u9sb 2022-08-17T12:28:27.4713641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61i2u9sb/_remote_module_non_scriptable.py 2022-08-17T12:28:27.4935391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2iz7cvss 2022-08-17T12:28:27.4936847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2iz7cvss/_remote_module_non_scriptable.py 2022-08-17T12:28:27.7236934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:27.7264801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:27.7295760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:27.7553630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:28.1110572Z skip: Need at least 2 CUDA devices (2.187s) 2022-08-17T12:28:28.1110860Z 2022-08-17T12:28:28.1111406Z ---------------------------------------------------------------------- 2022-08-17T12:28:28.1111757Z Ran 1 test in 2.187s 2022-08-17T12:28:28.1111878Z 2022-08-17T12:28:28.1111954Z OK (skipped=1) 2022-08-17T12:28:28.1112067Z 2022-08-17T12:28:28.1112156Z Generating XML reports... 2022-08-17T12:28:28.1165488Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122825.xml 2022-08-17T12:28:29.3927700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:29.3928099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:29.3930686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:29.3931056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:29.5450794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66_2i_r3 2022-08-17T12:28:29.5454116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66_2i_r3/_remote_module_non_scriptable.py 2022-08-17T12:28:29.7987821Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:29.7998924Z 2022-08-17T12:28:29.7999047Z Running tests... 2022-08-17T12:28:29.7999662Z ---------------------------------------------------------------------- 2022-08-17T12:28:30.0576836Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24895 2022-08-17T12:28:30.0603799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24896 2022-08-17T12:28:30.0631551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24897 2022-08-17T12:28:30.0661627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24898 2022-08-17T12:28:31.1065301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:31.1065930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:31.1068490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:31.1069086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:31.1228313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:31.1228942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:31.1232652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:31.1233336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:31.1695928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:31.1696614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:31.1698177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:31.1699144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:31.1846897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:31.1847440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:31.1851400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:31.1852063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:31.2835191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzkp6uyq 2022-08-17T12:28:31.2837318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzkp6uyq/_remote_module_non_scriptable.py 2022-08-17T12:28:31.2838088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcma_ebik 2022-08-17T12:28:31.2839347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcma_ebik/_remote_module_non_scriptable.py 2022-08-17T12:28:31.3282052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1s3adk0 2022-08-17T12:28:31.3282847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1s3adk0/_remote_module_non_scriptable.py 2022-08-17T12:28:31.3438870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiea0fi5u 2022-08-17T12:28:31.3439682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiea0fi5u/_remote_module_non_scriptable.py 2022-08-17T12:28:31.5422616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:31.5423675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:31.5882266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:31.6019309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:32.2719864Z ok (2.472s) 2022-08-17T12:28:32.2720108Z 2022-08-17T12:28:32.2720590Z ---------------------------------------------------------------------- 2022-08-17T12:28:32.2721009Z Ran 1 test in 2.472s 2022-08-17T12:28:32.2721193Z 2022-08-17T12:28:32.2721273Z OK 2022-08-17T12:28:32.2721424Z 2022-08-17T12:28:32.2721576Z Generating XML reports... 2022-08-17T12:28:32.2756921Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122829.xml 2022-08-17T12:28:33.5720615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:33.5721293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:33.5724334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:33.5724823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:33.7223332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamfm1lzf 2022-08-17T12:28:33.7226074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamfm1lzf/_remote_module_non_scriptable.py 2022-08-17T12:28:33.9802760Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:33.9813060Z 2022-08-17T12:28:33.9813211Z Running tests... 2022-08-17T12:28:33.9813802Z ---------------------------------------------------------------------- 2022-08-17T12:28:34.2444720Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25118 2022-08-17T12:28:34.2472680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25119 2022-08-17T12:28:34.2500632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25120 2022-08-17T12:28:34.2533379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25121 2022-08-17T12:28:35.3456803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:35.3457433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:35.3460642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:35.3461357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:35.3756725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:35.3757411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:35.3760157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:35.3760840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:35.4031888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:35.4032519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:35.4034028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:35.4034665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:35.4100518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:35.4101237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:35.4105138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:35.4105779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:35.5167474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1x_ucen 2022-08-17T12:28:35.5168263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1x_ucen/_remote_module_non_scriptable.py 2022-08-17T12:28:35.5356888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw5y6qud 2022-08-17T12:28:35.5358178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw5y6qud/_remote_module_non_scriptable.py 2022-08-17T12:28:35.5616511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxty6iyuf 2022-08-17T12:28:35.5617290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxty6iyuf/_remote_module_non_scriptable.py 2022-08-17T12:28:35.5674675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp107n09do 2022-08-17T12:28:35.5675711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp107n09do/_remote_module_non_scriptable.py 2022-08-17T12:28:35.7775025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:35.7983404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:35.8222088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:35.8454110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:36.1583516Z skip: Need at least 1 CUDA device (2.177s) 2022-08-17T12:28:36.1583859Z 2022-08-17T12:28:36.1584238Z ---------------------------------------------------------------------- 2022-08-17T12:28:36.1584512Z Ran 1 test in 2.177s 2022-08-17T12:28:36.1584621Z 2022-08-17T12:28:36.1584700Z OK (skipped=1) 2022-08-17T12:28:36.1584814Z 2022-08-17T12:28:36.1586437Z Generating XML reports... 2022-08-17T12:28:36.1621170Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122833.xml 2022-08-17T12:28:37.4427845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:37.4428652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:37.4431181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:37.4431794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:37.5932141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx40o7wth 2022-08-17T12:28:37.5935586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx40o7wth/_remote_module_non_scriptable.py 2022-08-17T12:28:37.8493578Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:37.8504492Z 2022-08-17T12:28:37.8504908Z Running tests... 2022-08-17T12:28:37.8505444Z ---------------------------------------------------------------------- 2022-08-17T12:28:38.1110365Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25173 2022-08-17T12:28:38.1138126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25174 2022-08-17T12:28:38.1166558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25175 2022-08-17T12:28:38.1197364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25176 2022-08-17T12:28:39.1075093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:39.1075759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:39.1079257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:39.1079957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:39.1317986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:39.1318685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:39.1322220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:39.1322845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:39.1399456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:39.1400456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:39.1403408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:39.1404049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:39.1494568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:39.1495274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:39.1497852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:39.1498532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:39.2682447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u7sx452 2022-08-17T12:28:39.2683248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u7sx452/_remote_module_non_scriptable.py 2022-08-17T12:28:39.2919270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf94hcoyu 2022-08-17T12:28:39.2920360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf94hcoyu/_remote_module_non_scriptable.py 2022-08-17T12:28:39.2998282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlfrg68h 2022-08-17T12:28:39.2999326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlfrg68h/_remote_module_non_scriptable.py 2022-08-17T12:28:39.3095192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzemkmxe1 2022-08-17T12:28:39.3096579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzemkmxe1/_remote_module_non_scriptable.py 2022-08-17T12:28:39.5510711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:39.5575879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:39.5614860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:39.5753583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:39.9248549Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T12:28:39.9248906Z 2022-08-17T12:28:39.9249359Z ---------------------------------------------------------------------- 2022-08-17T12:28:39.9249624Z Ran 1 test in 2.074s 2022-08-17T12:28:39.9249747Z 2022-08-17T12:28:39.9249821Z OK (skipped=1) 2022-08-17T12:28:39.9249946Z 2022-08-17T12:28:39.9250081Z Generating XML reports... 2022-08-17T12:28:39.9286949Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122837.xml 2022-08-17T12:28:41.2540878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:41.2541364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:41.2543240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:41.2543691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:41.4048037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50agvnmz 2022-08-17T12:28:41.4051201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50agvnmz/_remote_module_non_scriptable.py 2022-08-17T12:28:41.6669310Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:41.6679794Z 2022-08-17T12:28:41.6679933Z Running tests... 2022-08-17T12:28:41.6680345Z ---------------------------------------------------------------------- 2022-08-17T12:28:41.9384423Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25228 2022-08-17T12:28:41.9413660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25229 2022-08-17T12:28:41.9443290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25230 2022-08-17T12:28:41.9474434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25231 2022-08-17T12:28:43.0194003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:43.0194697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:43.0197353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:43.0198034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:43.0497645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:43.0498311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:43.0500321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:43.0500874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:43.0551582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:43.0552257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:43.0555382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:43.0556045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:43.1290339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:43.1290990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:43.1294227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:43.1294857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:43.1768579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojtx3_3c 2022-08-17T12:28:43.1769435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojtx3_3c/_remote_module_non_scriptable.py 2022-08-17T12:28:43.2082579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsatpt36j 2022-08-17T12:28:43.2083553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsatpt36j/_remote_module_non_scriptable.py 2022-08-17T12:28:43.2163540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjim82ew 2022-08-17T12:28:43.2164386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjim82ew/_remote_module_non_scriptable.py 2022-08-17T12:28:43.2867938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd7vokup 2022-08-17T12:28:43.2868918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd7vokup/_remote_module_non_scriptable.py 2022-08-17T12:28:43.4385847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:43.4706000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:43.4811975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:43.5507803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:43.8526103Z skip: Need at least 2 CUDA devices (2.184s) 2022-08-17T12:28:43.8526304Z 2022-08-17T12:28:43.8526628Z ---------------------------------------------------------------------- 2022-08-17T12:28:43.8526881Z Ran 1 test in 2.184s 2022-08-17T12:28:43.8527258Z 2022-08-17T12:28:43.8527335Z OK (skipped=1) 2022-08-17T12:28:43.8527450Z 2022-08-17T12:28:43.8527540Z Generating XML reports... 2022-08-17T12:28:43.8563630Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122841.xml 2022-08-17T12:28:45.1592677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:45.1593076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:45.1595607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:45.1596105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:45.3124392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4ratroz 2022-08-17T12:28:45.3127627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4ratroz/_remote_module_non_scriptable.py 2022-08-17T12:28:45.5727396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:45.5738192Z 2022-08-17T12:28:45.5738326Z Running tests... 2022-08-17T12:28:45.5738957Z ---------------------------------------------------------------------- 2022-08-17T12:28:45.8289124Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s) 2022-08-17T12:28:45.8290077Z 2022-08-17T12:28:45.8290326Z ---------------------------------------------------------------------- 2022-08-17T12:28:45.8290585Z Ran 1 test in 0.255s 2022-08-17T12:28:45.8290704Z 2022-08-17T12:28:45.8290767Z OK (skipped=1) 2022-08-17T12:28:45.8290888Z 2022-08-17T12:28:45.8290979Z Generating XML reports... 2022-08-17T12:28:45.8316132Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122845.xml 2022-08-17T12:28:47.0779022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:47.0779724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:47.0782750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:47.0783391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:47.2307730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgronn3vs 2022-08-17T12:28:47.2311342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgronn3vs/_remote_module_non_scriptable.py 2022-08-17T12:28:47.4907563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:47.4919189Z 2022-08-17T12:28:47.4919686Z Running tests... 2022-08-17T12:28:47.4920100Z ---------------------------------------------------------------------- 2022-08-17T12:28:47.7558262Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25293 2022-08-17T12:28:47.7588588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25294 2022-08-17T12:28:47.7618005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25295 2022-08-17T12:28:47.7648783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25296 2022-08-17T12:28:48.7627279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:48.7627992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:48.7631023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:48.7631730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:48.7643261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:48.7643862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:48.7648521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:48.7649157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:48.8161182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:48.8161863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:48.8164979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:48.8165588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:48.8352809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:48.8353500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:48.8357748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:48.8358441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:48.9297218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q465e26 2022-08-17T12:28:48.9298059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q465e26/_remote_module_non_scriptable.py 2022-08-17T12:28:48.9411545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3fss_sj 2022-08-17T12:28:48.9412514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3fss_sj/_remote_module_non_scriptable.py 2022-08-17T12:28:48.9792793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprniff71k 2022-08-17T12:28:48.9793680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprniff71k/_remote_module_non_scriptable.py 2022-08-17T12:28:48.9985899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppktqo2qu 2022-08-17T12:28:48.9987176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppktqo2qu/_remote_module_non_scriptable.py 2022-08-17T12:28:49.1998823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:49.2081327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:49.2478034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:49.2689523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:49.5698775Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:28:49.5699056Z 2022-08-17T12:28:49.5699605Z ---------------------------------------------------------------------- 2022-08-17T12:28:49.5699936Z Ran 1 test in 2.078s 2022-08-17T12:28:49.5700056Z 2022-08-17T12:28:49.5700118Z OK (skipped=1) 2022-08-17T12:28:49.5700237Z 2022-08-17T12:28:49.5700326Z Generating XML reports... 2022-08-17T12:28:49.5736402Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122847.xml 2022-08-17T12:28:50.9044550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:50.9045285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:50.9048469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:50.9049347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:51.0650657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wjk4yqp 2022-08-17T12:28:51.0654275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wjk4yqp/_remote_module_non_scriptable.py 2022-08-17T12:28:51.3269335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:51.3280789Z 2022-08-17T12:28:51.3281257Z Running tests... 2022-08-17T12:28:51.3281713Z ---------------------------------------------------------------------- 2022-08-17T12:28:51.5928837Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25348 2022-08-17T12:28:51.5957194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25349 2022-08-17T12:28:51.5985448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25350 2022-08-17T12:28:51.6016039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25351 2022-08-17T12:28:52.5764447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:52.5765122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:52.5767457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:52.5768100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:52.6366516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:52.6367218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:52.6370313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:52.6370994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:52.6375146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:52.6375766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:52.6379754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:52.6380384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:52.6547106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:52.6547765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:52.6550872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:52.6551568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:52.7355325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqi15a5oj 2022-08-17T12:28:52.7356178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqi15a5oj/_remote_module_non_scriptable.py 2022-08-17T12:28:52.7997333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxmfvjko 2022-08-17T12:28:52.7998137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxmfvjko/_remote_module_non_scriptable.py 2022-08-17T12:28:52.7999083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg0fdexj 2022-08-17T12:28:52.8002423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg0fdexj/_remote_module_non_scriptable.py 2022-08-17T12:28:52.8172623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvoqhw49g 2022-08-17T12:28:52.8173656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvoqhw49g/_remote_module_non_scriptable.py 2022-08-17T12:28:52.9992982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:53.0677712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:53.0834254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:53.0834865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:53.4067104Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:28:53.4067406Z 2022-08-17T12:28:53.4067907Z ---------------------------------------------------------------------- 2022-08-17T12:28:53.4068447Z Ran 1 test in 2.079s 2022-08-17T12:28:53.4068680Z 2022-08-17T12:28:53.4068815Z OK (skipped=1) 2022-08-17T12:28:53.4068973Z 2022-08-17T12:28:53.4069065Z Generating XML reports... 2022-08-17T12:28:53.4104940Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122851.xml 2022-08-17T12:28:54.7230693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:54.7231306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:54.7233926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:54.7234320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:54.8777017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sl9aeff 2022-08-17T12:28:54.8780426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sl9aeff/_remote_module_non_scriptable.py 2022-08-17T12:28:55.1381452Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:55.1392862Z 2022-08-17T12:28:55.1393303Z Running tests... 2022-08-17T12:28:55.1393965Z ---------------------------------------------------------------------- 2022-08-17T12:28:55.4004309Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25403 2022-08-17T12:28:55.4032641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25404 2022-08-17T12:28:55.4060672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25405 2022-08-17T12:28:55.4091398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25406 2022-08-17T12:28:56.3974350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:56.3974997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:56.3977929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:56.3978607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:56.4092840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:56.4093435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:56.4096892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:56.4097558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:56.4101806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:56.4102450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:56.4106986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:56.4107927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:56.4734371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:56.4734935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:56.4737649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:56.4738201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:56.5563762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfg5v_a8 2022-08-17T12:28:56.5564620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfg5v_a8/_remote_module_non_scriptable.py 2022-08-17T12:28:56.5683128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeeznxtz 2022-08-17T12:28:56.5683882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7oo27zi 2022-08-17T12:28:56.5684478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeeznxtz/_remote_module_non_scriptable.py 2022-08-17T12:28:56.5685791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7oo27zi/_remote_module_non_scriptable.py 2022-08-17T12:28:56.6309479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfx6t5j7r 2022-08-17T12:28:56.6310210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfx6t5j7r/_remote_module_non_scriptable.py 2022-08-17T12:28:56.8189350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:28:56.8293656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:28:56.8294264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:28:56.8960815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:28:57.2141288Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T12:28:57.2141585Z 2022-08-17T12:28:57.2142132Z ---------------------------------------------------------------------- 2022-08-17T12:28:57.2142458Z Ran 1 test in 2.075s 2022-08-17T12:28:57.2142598Z 2022-08-17T12:28:57.2142662Z OK (skipped=1) 2022-08-17T12:28:57.2142776Z 2022-08-17T12:28:57.2142867Z Generating XML reports... 2022-08-17T12:28:57.2178625Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122855.xml 2022-08-17T12:28:58.5216782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:28:58.5217183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:28:58.5219224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:28:58.5219756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:28:58.6744727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppg8whlbq 2022-08-17T12:28:58.6748078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppg8whlbq/_remote_module_non_scriptable.py 2022-08-17T12:28:58.9343385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:28:58.9354192Z 2022-08-17T12:28:58.9354488Z Running tests... 2022-08-17T12:28:58.9355092Z ---------------------------------------------------------------------- 2022-08-17T12:28:59.2033093Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25458 2022-08-17T12:28:59.2062072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25459 2022-08-17T12:28:59.2092326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25460 2022-08-17T12:28:59.2124916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25461 2022-08-17T12:29:00.2534225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:00.2534890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:00.2537818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:00.2538504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:00.3186219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:00.3186900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:00.3189595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:00.3190234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:00.3218119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:00.3219666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:00.3222152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:00.3222804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:00.3350770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:00.3351402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:00.3353314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:00.3353906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:00.4181363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lymnchr 2022-08-17T12:29:00.4182147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lymnchr/_remote_module_non_scriptable.py 2022-08-17T12:29:00.4779129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3oixt2v 2022-08-17T12:29:00.4779923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3oixt2v/_remote_module_non_scriptable.py 2022-08-17T12:29:00.4968642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e6a22k2 2022-08-17T12:29:00.4969337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e6a22k2/_remote_module_non_scriptable.py 2022-08-17T12:29:00.5025692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7222nqwn 2022-08-17T12:29:00.5027207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7222nqwn/_remote_module_non_scriptable.py 2022-08-17T12:29:00.6837010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:00.7398409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:00.7616837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:00.8001148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:01.1177195Z skip: Need at least 2 CUDA devices (2.182s) 2022-08-17T12:29:01.1177502Z 2022-08-17T12:29:01.1177990Z ---------------------------------------------------------------------- 2022-08-17T12:29:01.1178398Z Ran 1 test in 2.182s 2022-08-17T12:29:01.1178583Z 2022-08-17T12:29:01.1178697Z OK (skipped=1) 2022-08-17T12:29:01.1178860Z 2022-08-17T12:29:01.1178994Z Generating XML reports... 2022-08-17T12:29:01.1215139Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122858.xml 2022-08-17T12:29:02.4307622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:02.4308073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:02.4310933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:02.4311361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:02.5835272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa1ylmog 2022-08-17T12:29:02.5838178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa1ylmog/_remote_module_non_scriptable.py 2022-08-17T12:29:02.8444444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:02.8455965Z 2022-08-17T12:29:02.8456386Z Running tests... 2022-08-17T12:29:02.8456781Z ---------------------------------------------------------------------- 2022-08-17T12:29:03.1075936Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25513 2022-08-17T12:29:03.1105781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25514 2022-08-17T12:29:03.1134317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25515 2022-08-17T12:29:03.1165433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25516 2022-08-17T12:29:04.1746113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:04.1746768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:04.1749840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:04.1750476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:04.2005147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:04.2005797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:04.2008991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:04.2009595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:04.2243698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:04.2244362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:04.2247742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:04.2248439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:04.2473138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:04.2473869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:04.2477179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:04.2477813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:04.3336926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmiscbfg 2022-08-17T12:29:04.3337734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmiscbfg/_remote_module_non_scriptable.py 2022-08-17T12:29:04.3599681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm94m56k 2022-08-17T12:29:04.3600698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm94m56k/_remote_module_non_scriptable.py 2022-08-17T12:29:04.3868661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9k95lh2d 2022-08-17T12:29:04.3870062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9k95lh2d/_remote_module_non_scriptable.py 2022-08-17T12:29:04.4047455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzsc3dya 2022-08-17T12:29:04.4048674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzsc3dya/_remote_module_non_scriptable.py 2022-08-17T12:29:04.6209864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:04.6243627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:04.6508024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:04.6661576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:05.0216779Z skip: Need at least 2 CUDA devices (2.176s) 2022-08-17T12:29:05.0216971Z 2022-08-17T12:29:05.0217416Z ---------------------------------------------------------------------- 2022-08-17T12:29:05.0217724Z Ran 1 test in 2.176s 2022-08-17T12:29:05.0217851Z 2022-08-17T12:29:05.0217929Z OK (skipped=1) 2022-08-17T12:29:05.0218267Z 2022-08-17T12:29:05.0218360Z Generating XML reports... 2022-08-17T12:29:05.0254063Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122902.xml 2022-08-17T12:29:06.3115534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:06.3115912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:06.3118691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:06.3119118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:06.4637813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90w_jma7 2022-08-17T12:29:06.4641260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90w_jma7/_remote_module_non_scriptable.py 2022-08-17T12:29:06.7235827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:06.7246504Z 2022-08-17T12:29:06.7246596Z Running tests... 2022-08-17T12:29:06.7247329Z ---------------------------------------------------------------------- 2022-08-17T12:29:06.9916917Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25568 2022-08-17T12:29:06.9945229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25569 2022-08-17T12:29:06.9973992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25570 2022-08-17T12:29:07.0004441Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25571 2022-08-17T12:29:07.9848272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:07.9848945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:07.9852059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:07.9852717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:08.0093698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:08.0094354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:08.0097608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:08.0098558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:08.0354553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:08.0355260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:08.0358886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:08.0359665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:08.0560667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:08.0561313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:08.0562684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:08.0563296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:08.1454910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_h6fjp0 2022-08-17T12:29:08.1455841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_h6fjp0/_remote_module_non_scriptable.py 2022-08-17T12:29:08.1694681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9anrfhi 2022-08-17T12:29:08.1695534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9anrfhi/_remote_module_non_scriptable.py 2022-08-17T12:29:08.1943133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5q5gqy2 2022-08-17T12:29:08.1944397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5q5gqy2/_remote_module_non_scriptable.py 2022-08-17T12:29:08.2132876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdasahair 2022-08-17T12:29:08.2158785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdasahair/_remote_module_non_scriptable.py 2022-08-17T12:29:08.4092738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:08.4292390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:08.4540304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:08.4760673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:08.8054762Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T12:29:08.8055085Z 2022-08-17T12:29:08.8055609Z ---------------------------------------------------------------------- 2022-08-17T12:29:08.8056069Z Ran 1 test in 2.081s 2022-08-17T12:29:08.8056281Z 2022-08-17T12:29:08.8056414Z OK (skipped=1) 2022-08-17T12:29:08.8056615Z 2022-08-17T12:29:08.8056769Z Generating XML reports... 2022-08-17T12:29:08.8091940Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122906.xml 2022-08-17T12:29:10.1275980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:10.1276376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:10.1278813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:10.1279180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:10.2770172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkp34m7pe 2022-08-17T12:29:10.2773187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkp34m7pe/_remote_module_non_scriptable.py 2022-08-17T12:29:10.5325715Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:10.5336228Z 2022-08-17T12:29:10.5336321Z Running tests... 2022-08-17T12:29:10.5337298Z ---------------------------------------------------------------------- 2022-08-17T12:29:10.7953859Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25623 2022-08-17T12:29:10.7981104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25624 2022-08-17T12:29:10.8012595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25625 2022-08-17T12:29:10.8042707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25626 2022-08-17T12:29:11.8399235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:11.8399888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:11.8403224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:11.8403923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:11.8760346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:11.8760958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:11.8764265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:11.8764952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:11.9033972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:11.9034632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:11.9036729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:11.9037399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:11.9053134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:11.9053799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:11.9057928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:11.9058615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:11.9972660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zvvzwbo 2022-08-17T12:29:11.9973391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zvvzwbo/_remote_module_non_scriptable.py 2022-08-17T12:29:12.0327367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp387kh8rc 2022-08-17T12:29:12.0328152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp387kh8rc/_remote_module_non_scriptable.py 2022-08-17T12:29:12.0637413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjljqhyu 2022-08-17T12:29:12.0638176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmvih7x4 2022-08-17T12:29:12.0638843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjljqhyu/_remote_module_non_scriptable.py 2022-08-17T12:29:12.0641519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmvih7x4/_remote_module_non_scriptable.py 2022-08-17T12:29:12.2572765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:12.2882480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:12.3234922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:12.3256961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:12.6093523Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T12:29:12.6094101Z 2022-08-17T12:29:12.6094548Z ---------------------------------------------------------------------- 2022-08-17T12:29:12.6094829Z Ran 1 test in 2.076s 2022-08-17T12:29:12.6094936Z 2022-08-17T12:29:12.6095012Z OK (skipped=1) 2022-08-17T12:29:12.6095125Z 2022-08-17T12:29:12.6095213Z Generating XML reports... 2022-08-17T12:29:12.6130128Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122910.xml 2022-08-17T12:29:13.8877655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:13.8878062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:13.8880896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:13.8881314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:14.0406683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2q52x8z 2022-08-17T12:29:14.0411441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2q52x8z/_remote_module_non_scriptable.py 2022-08-17T12:29:14.2995560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:14.3006421Z 2022-08-17T12:29:14.3006777Z Running tests... 2022-08-17T12:29:14.3007447Z ---------------------------------------------------------------------- 2022-08-17T12:29:14.5639075Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25678 2022-08-17T12:29:14.5667348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25679 2022-08-17T12:29:14.5696784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25680 2022-08-17T12:29:14.5727504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25681 2022-08-17T12:29:15.5880359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:15.5880974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:15.5884212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:15.5884833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:15.6334192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:15.6334880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:15.6337388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:15.6338043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:15.6572845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:15.6573478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:15.6576261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:15.6576854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:15.6614895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:15.6615498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:15.6619839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:15.6620460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:15.7448592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy7fbbl_ 2022-08-17T12:29:15.7450201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy7fbbl_/_remote_module_non_scriptable.py 2022-08-17T12:29:15.7923185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug4gs0p5 2022-08-17T12:29:15.7923988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug4gs0p5/_remote_module_non_scriptable.py 2022-08-17T12:29:15.8147040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuo1iva4h 2022-08-17T12:29:15.8147757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuo1iva4h/_remote_module_non_scriptable.py 2022-08-17T12:29:15.8194529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0ew1unk 2022-08-17T12:29:15.8197440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0ew1unk/_remote_module_non_scriptable.py 2022-08-17T12:29:16.0059387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:16.0747232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:16.0782725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:16.0983076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:16.3778576Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T12:29:16.3778908Z 2022-08-17T12:29:16.3779308Z ---------------------------------------------------------------------- 2022-08-17T12:29:16.3779562Z Ran 1 test in 2.077s 2022-08-17T12:29:16.3779697Z 2022-08-17T12:29:16.3779776Z OK (skipped=1) 2022-08-17T12:29:16.3779891Z 2022-08-17T12:29:16.3779983Z Generating XML reports... 2022-08-17T12:29:16.3818034Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122914.xml 2022-08-17T12:29:17.6842428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:17.6842822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:17.6845589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:17.6846282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:17.8362239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn0vvmo4 2022-08-17T12:29:17.8365695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn0vvmo4/_remote_module_non_scriptable.py 2022-08-17T12:29:18.0967903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:18.0978261Z 2022-08-17T12:29:18.0978472Z Running tests... 2022-08-17T12:29:18.0978870Z ---------------------------------------------------------------------- 2022-08-17T12:29:18.3617971Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25733 2022-08-17T12:29:18.3645674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25734 2022-08-17T12:29:18.3675148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25735 2022-08-17T12:29:18.3705604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25736 2022-08-17T12:29:19.4057009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:19.4057671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:19.4061021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:19.4061692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:19.4171367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:19.4172009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:19.4175467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:19.4176168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:19.4475398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:19.4476089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:19.4479094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:19.4479693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:19.4868206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:19.4869005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:19.4872146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:19.4873179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:19.5652127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphtlye49 2022-08-17T12:29:19.5652945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphtlye49/_remote_module_non_scriptable.py 2022-08-17T12:29:19.5789319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_600gasn 2022-08-17T12:29:19.5790119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_600gasn/_remote_module_non_scriptable.py 2022-08-17T12:29:19.6076895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau8y59es 2022-08-17T12:29:19.6077730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau8y59es/_remote_module_non_scriptable.py 2022-08-17T12:29:19.6435882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3s5ei3n 2022-08-17T12:29:19.6436712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3s5ei3n/_remote_module_non_scriptable.py 2022-08-17T12:29:19.8238983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:19.8435362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:19.8687202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:19.9072623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:20.1755488Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T12:29:20.1755797Z 2022-08-17T12:29:20.1756347Z ---------------------------------------------------------------------- 2022-08-17T12:29:20.1756642Z Ran 1 test in 2.078s 2022-08-17T12:29:20.1756762Z 2022-08-17T12:29:20.1756837Z OK (skipped=1) 2022-08-17T12:29:20.1756947Z 2022-08-17T12:29:20.1757034Z Generating XML reports... 2022-08-17T12:29:20.1792912Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122918.xml 2022-08-17T12:29:21.4608014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:21.4608402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:21.4610814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:21.4611275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:21.6102636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklsklslz 2022-08-17T12:29:21.6105623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklsklslz/_remote_module_non_scriptable.py 2022-08-17T12:29:21.8686733Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:21.8697662Z 2022-08-17T12:29:21.8697834Z Running tests... 2022-08-17T12:29:21.8698473Z ---------------------------------------------------------------------- 2022-08-17T12:29:22.1338834Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25788 2022-08-17T12:29:22.1367337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25789 2022-08-17T12:29:22.1395941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25790 2022-08-17T12:29:22.1426069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25791 2022-08-17T12:29:23.1338354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:23.1339022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:23.1342981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:23.1343651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:23.1380918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:23.1381543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:23.1386072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:23.1386724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:23.1462675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:23.1463359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:23.1466969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:23.1467636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:23.2160862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:23.2161613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:23.2163713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:23.2164434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:23.2981403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92hv2viu 2022-08-17T12:29:23.2982255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92hv2viu/_remote_module_non_scriptable.py 2022-08-17T12:29:23.3079306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrg2v72m 2022-08-17T12:29:23.3080391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrg2v72m/_remote_module_non_scriptable.py 2022-08-17T12:29:23.3109037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j3pxxxp 2022-08-17T12:29:23.3110413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j3pxxxp/_remote_module_non_scriptable.py 2022-08-17T12:29:23.3781174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vdj_ym2 2022-08-17T12:29:23.3782484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vdj_ym2/_remote_module_non_scriptable.py 2022-08-17T12:29:23.5593795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:23.5674639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:23.5703406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:23.6391165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:23.9477119Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:29:23.9477389Z 2022-08-17T12:29:23.9477714Z ---------------------------------------------------------------------- 2022-08-17T12:29:23.9477976Z Ran 1 test in 2.078s 2022-08-17T12:29:23.9478163Z 2022-08-17T12:29:23.9478241Z OK (skipped=1) 2022-08-17T12:29:23.9478353Z 2022-08-17T12:29:23.9478447Z Generating XML reports... 2022-08-17T12:29:23.9514677Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122921.xml 2022-08-17T12:29:25.2544937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:25.2545591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:25.2546745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:25.2547652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:25.4071732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq6p64x1 2022-08-17T12:29:25.4075055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq6p64x1/_remote_module_non_scriptable.py 2022-08-17T12:29:25.6644983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:25.6655977Z 2022-08-17T12:29:25.6656130Z Running tests... 2022-08-17T12:29:25.6656478Z ---------------------------------------------------------------------- 2022-08-17T12:29:25.9288025Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25843 2022-08-17T12:29:25.9318236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25844 2022-08-17T12:29:25.9347673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25845 2022-08-17T12:29:25.9379450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25846 2022-08-17T12:29:26.9704338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:26.9704967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:26.9707905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:26.9708622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:26.9981469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:26.9982156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:26.9985093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:26.9985782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:27.0071666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:27.0072323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:27.0075474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:27.0076198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:27.0275002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:27.0275894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:27.0277814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:27.0278408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:27.1321828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsgm18r4 2022-08-17T12:29:27.1322565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsgm18r4/_remote_module_non_scriptable.py 2022-08-17T12:29:27.1600381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpdu6hsb 2022-08-17T12:29:27.1601218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpdu6hsb/_remote_module_non_scriptable.py 2022-08-17T12:29:27.1677259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuprbt2r8 2022-08-17T12:29:27.1678818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuprbt2r8/_remote_module_non_scriptable.py 2022-08-17T12:29:27.1858095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsso8dxg 2022-08-17T12:29:27.1859314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsso8dxg/_remote_module_non_scriptable.py 2022-08-17T12:29:27.3991446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:27.4244988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:27.4438186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:27.4462176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:27.7429948Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T12:29:27.7430251Z 2022-08-17T12:29:27.7430666Z ---------------------------------------------------------------------- 2022-08-17T12:29:27.7430946Z Ran 1 test in 2.077s 2022-08-17T12:29:27.7431068Z 2022-08-17T12:29:27.7431144Z OK (skipped=1) 2022-08-17T12:29:27.7431257Z 2022-08-17T12:29:27.7431345Z Generating XML reports... 2022-08-17T12:29:27.7467001Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122925.xml 2022-08-17T12:29:29.0328590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:29.0329039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:29.0331236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:29.0331609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:29.1848061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dmp2h1z 2022-08-17T12:29:29.1851367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dmp2h1z/_remote_module_non_scriptable.py 2022-08-17T12:29:29.4416458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:29.4427144Z 2022-08-17T12:29:29.4427256Z Running tests... 2022-08-17T12:29:29.4428029Z ---------------------------------------------------------------------- 2022-08-17T12:29:29.7074275Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25898 2022-08-17T12:29:29.7103484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25899 2022-08-17T12:29:29.7132852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25900 2022-08-17T12:29:29.7163536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25901 2022-08-17T12:29:30.7715591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:30.7716523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:30.7719197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:30.7719810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:30.7751177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:30.7751812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:30.7755019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:30.7755706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:30.7991061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:30.7991775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:30.7993473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:30.7994450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:30.8335342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:30.8335981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:30.8339147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:30.8339848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:30.9306941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzz_8hvo 2022-08-17T12:29:30.9307776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzz_8hvo/_remote_module_non_scriptable.py 2022-08-17T12:29:30.9337812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlowciyl 2022-08-17T12:29:30.9338995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlowciyl/_remote_module_non_scriptable.py 2022-08-17T12:29:30.9565566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxp2wr0rn 2022-08-17T12:29:30.9566736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxp2wr0rn/_remote_module_non_scriptable.py 2022-08-17T12:29:30.9920987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96qu425g 2022-08-17T12:29:30.9921754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96qu425g/_remote_module_non_scriptable.py 2022-08-17T12:29:31.1912063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:31.1941759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:31.2125504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:31.2521147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:31.5213738Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:29:31.5214080Z 2022-08-17T12:29:31.5214533Z ---------------------------------------------------------------------- 2022-08-17T12:29:31.5214863Z Ran 1 test in 2.079s 2022-08-17T12:29:31.5214994Z 2022-08-17T12:29:31.5215059Z OK (skipped=1) 2022-08-17T12:29:31.5215174Z 2022-08-17T12:29:31.5215263Z Generating XML reports... 2022-08-17T12:29:31.5250801Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122929.xml 2022-08-17T12:29:32.8260731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:32.8261388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:32.8263694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:32.8264334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:32.9790025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnvbep_5 2022-08-17T12:29:32.9793789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnvbep_5/_remote_module_non_scriptable.py 2022-08-17T12:29:33.2395551Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:33.2406522Z 2022-08-17T12:29:33.2406677Z Running tests... 2022-08-17T12:29:33.2407330Z ---------------------------------------------------------------------- 2022-08-17T12:29:33.5073760Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25953 2022-08-17T12:29:33.5102920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25954 2022-08-17T12:29:33.5131669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25955 2022-08-17T12:29:33.5162649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25956 2022-08-17T12:29:34.6097927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:34.6098547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:34.6101313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:34.6101935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:34.6366415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:34.6367041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:34.6369588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:34.6370212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:34.6416792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:34.6417433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:34.6420929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:34.6421616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:34.6535119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:34.6535817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:34.6538607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:34.6539230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:34.7736649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wc5ow6q 2022-08-17T12:29:34.7737741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wc5ow6q/_remote_module_non_scriptable.py 2022-08-17T12:29:34.8020279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7b_hw44 2022-08-17T12:29:34.8021149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7b_hw44/_remote_module_non_scriptable.py 2022-08-17T12:29:34.8137678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax1ty86t 2022-08-17T12:29:34.8139102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax1ty86t/_remote_module_non_scriptable.py 2022-08-17T12:29:34.8170989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6alvry_r 2022-08-17T12:29:34.8172365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6alvry_r/_remote_module_non_scriptable.py 2022-08-17T12:29:35.0388430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:35.0624134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:35.0761992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:35.0788182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:35.4213875Z skip: Need at least 2 CUDA devices (2.180s) 2022-08-17T12:29:35.4214175Z 2022-08-17T12:29:35.4214679Z ---------------------------------------------------------------------- 2022-08-17T12:29:35.4215127Z Ran 1 test in 2.181s 2022-08-17T12:29:35.4215272Z 2022-08-17T12:29:35.4215349Z OK (skipped=1) 2022-08-17T12:29:35.4215464Z 2022-08-17T12:29:35.4215552Z Generating XML reports... 2022-08-17T12:29:35.4251728Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122933.xml 2022-08-17T12:29:36.7093952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:36.7094360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:36.7096710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:36.7097089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:36.8631125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph030j3mv 2022-08-17T12:29:36.8634072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph030j3mv/_remote_module_non_scriptable.py 2022-08-17T12:29:37.1231668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:37.1242223Z 2022-08-17T12:29:37.1242364Z Running tests... 2022-08-17T12:29:37.1243126Z ---------------------------------------------------------------------- 2022-08-17T12:29:37.3874152Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26008 2022-08-17T12:29:37.3902437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26009 2022-08-17T12:29:37.3931699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26010 2022-08-17T12:29:37.3964953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26011 2022-08-17T12:29:38.4138611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:38.4139306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:38.4141774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:38.4142380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:38.4144026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:38.4144669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:38.4153173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:38.4153819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:38.4154638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:38.4155542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:38.4156368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:38.4156995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:38.4355491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:38.4356144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:38.4358904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:38.4359508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:38.5763979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ksx6zu1 2022-08-17T12:29:38.5764760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ksx6zu1/_remote_module_non_scriptable.py 2022-08-17T12:29:38.5766283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4r5736_ 2022-08-17T12:29:38.5767002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne6fwgd8 2022-08-17T12:29:38.5770031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4r5736_/_remote_module_non_scriptable.py 2022-08-17T12:29:38.5771735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne6fwgd8/_remote_module_non_scriptable.py 2022-08-17T12:29:38.5950458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k5cyy8_ 2022-08-17T12:29:38.5951759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k5cyy8_/_remote_module_non_scriptable.py 2022-08-17T12:29:38.8401217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:38.8413840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:38.8450825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:38.8577288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:39.2014974Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T12:29:39.2015275Z 2022-08-17T12:29:39.2015728Z ---------------------------------------------------------------------- 2022-08-17T12:29:39.2015989Z Ran 1 test in 2.077s 2022-08-17T12:29:39.2016120Z 2022-08-17T12:29:39.2016199Z OK (skipped=1) 2022-08-17T12:29:39.2016313Z 2022-08-17T12:29:39.2016389Z Generating XML reports... 2022-08-17T12:29:39.2052001Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122937.xml 2022-08-17T12:29:40.5110938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:40.5111341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:40.5113882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:40.5114263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:40.6608482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5z_4noq 2022-08-17T12:29:40.6611517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5z_4noq/_remote_module_non_scriptable.py 2022-08-17T12:29:40.9170983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:40.9181606Z 2022-08-17T12:29:40.9181737Z Running tests... 2022-08-17T12:29:40.9182126Z ---------------------------------------------------------------------- 2022-08-17T12:29:41.1843408Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26063 2022-08-17T12:29:41.1872124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26064 2022-08-17T12:29:41.1902195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26065 2022-08-17T12:29:41.1933369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26066 2022-08-17T12:29:42.2564454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:42.2565094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:42.2567357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:42.2567935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:42.3330869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:42.3331622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:42.3334353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:42.3335093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:42.3346673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:42.3347234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:42.3350908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:42.3351522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:42.3492841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:42.3493501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:42.3496211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:42.3496990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:42.4164935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnbi6fjd 2022-08-17T12:29:42.4165720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnbi6fjd/_remote_module_non_scriptable.py 2022-08-17T12:29:42.4937450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt147sgl 2022-08-17T12:29:42.4938462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt147sgl/_remote_module_non_scriptable.py 2022-08-17T12:29:42.4943542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uutemyb 2022-08-17T12:29:42.4945908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uutemyb/_remote_module_non_scriptable.py 2022-08-17T12:29:42.5108404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqmrmrdb 2022-08-17T12:29:42.5110649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqmrmrdb/_remote_module_non_scriptable.py 2022-08-17T12:29:42.6813957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:42.7584715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:42.7596709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:42.7779650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:43.0985639Z skip: Need at least 2 CUDA devices (2.180s) 2022-08-17T12:29:43.0985984Z 2022-08-17T12:29:43.0986437Z ---------------------------------------------------------------------- 2022-08-17T12:29:43.0986684Z Ran 1 test in 2.180s 2022-08-17T12:29:43.0986804Z 2022-08-17T12:29:43.0987130Z OK (skipped=1) 2022-08-17T12:29:43.0987245Z 2022-08-17T12:29:43.0987336Z Generating XML reports... 2022-08-17T12:29:43.1022389Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122940.xml 2022-08-17T12:29:44.4016946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:44.4017612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:44.4018608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:44.4019228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:44.5544051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhurq_cc 2022-08-17T12:29:44.5546538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhurq_cc/_remote_module_non_scriptable.py 2022-08-17T12:29:44.8146468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:44.8157765Z 2022-08-17T12:29:44.8158171Z Running tests... 2022-08-17T12:29:44.8158836Z ---------------------------------------------------------------------- 2022-08-17T12:29:45.0830375Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26118 2022-08-17T12:29:45.0858520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26119 2022-08-17T12:29:45.0889589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26120 2022-08-17T12:29:45.0924877Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26121 2022-08-17T12:29:46.0746352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:46.0747044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:46.0749881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:46.0750494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:46.0838061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:46.0838722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:46.0842328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:46.0842958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:46.1113598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:46.1114243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:46.1117369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:46.1117997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:46.1246470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:46.1247173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:46.1249675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:46.1250349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:46.2353873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n34g8np 2022-08-17T12:29:46.2355181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n34g8np/_remote_module_non_scriptable.py 2022-08-17T12:29:46.2460584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk93clgg 2022-08-17T12:29:46.2461861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk93clgg/_remote_module_non_scriptable.py 2022-08-17T12:29:46.2886764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97e020rf 2022-08-17T12:29:46.2887534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97e020rf/_remote_module_non_scriptable.py 2022-08-17T12:29:46.2910879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesk8s79m 2022-08-17T12:29:46.2912157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesk8s79m/_remote_module_non_scriptable.py 2022-08-17T12:29:46.4986617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:46.5104661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:46.5537984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:46.5538793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:46.8973162Z skip: Need at least 2 CUDA devices (2.081s) 2022-08-17T12:29:46.8973368Z 2022-08-17T12:29:46.8974052Z ---------------------------------------------------------------------- 2022-08-17T12:29:46.8974337Z Ran 1 test in 2.081s 2022-08-17T12:29:46.8974457Z 2022-08-17T12:29:46.8974535Z OK (skipped=1) 2022-08-17T12:29:46.8974635Z 2022-08-17T12:29:46.8974729Z Generating XML reports... 2022-08-17T12:29:46.9010320Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122944.xml 2022-08-17T12:29:48.1954983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:48.1955383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:48.1957218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:48.1957575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:48.3468643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoql7wh4t 2022-08-17T12:29:48.3472217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoql7wh4t/_remote_module_non_scriptable.py 2022-08-17T12:29:48.6072801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:48.6082735Z 2022-08-17T12:29:48.6082857Z Running tests... 2022-08-17T12:29:48.6083333Z ---------------------------------------------------------------------- 2022-08-17T12:29:48.8739292Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26173 2022-08-17T12:29:48.8767234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26174 2022-08-17T12:29:48.8796564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26175 2022-08-17T12:29:48.8827064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26176 2022-08-17T12:29:49.9355541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:49.9356238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:49.9358986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:49.9359689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:49.9857671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:49.9858230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:49.9860478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:49.9861012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:49.9966708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:49.9967278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:49.9970794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:49.9971464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:50.0157739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:50.0158435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:50.0160963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:50.0161665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:50.0948405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qj2c0xg 2022-08-17T12:29:50.0949772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qj2c0xg/_remote_module_non_scriptable.py 2022-08-17T12:29:50.1448850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthcobhjh 2022-08-17T12:29:50.1449842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthcobhjh/_remote_module_non_scriptable.py 2022-08-17T12:29:50.1574964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnv2gtbw 2022-08-17T12:29:50.1577143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnv2gtbw/_remote_module_non_scriptable.py 2022-08-17T12:29:50.1733032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9f0eua0g 2022-08-17T12:29:50.1734008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9f0eua0g/_remote_module_non_scriptable.py 2022-08-17T12:29:50.3571535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:50.4192902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:50.4208333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:50.4568358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:50.7878423Z skip: Need at least 2 CUDA devices (2.179s) 2022-08-17T12:29:50.7878728Z 2022-08-17T12:29:50.7879178Z ---------------------------------------------------------------------- 2022-08-17T12:29:50.7879571Z Ran 1 test in 2.180s 2022-08-17T12:29:50.7879757Z 2022-08-17T12:29:50.7879882Z OK (skipped=1) 2022-08-17T12:29:50.7880051Z 2022-08-17T12:29:50.7880225Z Generating XML reports... 2022-08-17T12:29:50.7917450Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122948.xml 2022-08-17T12:29:52.0724312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:52.0724711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:52.0727172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:52.0727783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:52.2226480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96d7r0n6 2022-08-17T12:29:52.2229658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96d7r0n6/_remote_module_non_scriptable.py 2022-08-17T12:29:52.4826903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:52.4837409Z 2022-08-17T12:29:52.4837562Z Running tests... 2022-08-17T12:29:52.4838133Z ---------------------------------------------------------------------- 2022-08-17T12:29:52.7460431Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26228 2022-08-17T12:29:52.7488086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26229 2022-08-17T12:29:52.7516068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26230 2022-08-17T12:29:52.7548050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26231 2022-08-17T12:29:53.8057979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:53.8058639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:53.8061471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:53.8062145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:53.8177078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:53.8177663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:53.8179888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:53.8180521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:53.8425553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:53.8426228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:53.8429075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:53.8429803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:53.8532771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:53.8533370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:53.8536507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:53.8537205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:53.9676394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vlvd9c_ 2022-08-17T12:29:53.9677578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vlvd9c_/_remote_module_non_scriptable.py 2022-08-17T12:29:53.9784743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yu_ztmn 2022-08-17T12:29:53.9787312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yu_ztmn/_remote_module_non_scriptable.py 2022-08-17T12:29:54.0040744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiczi03ej 2022-08-17T12:29:54.0041532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiczi03ej/_remote_module_non_scriptable.py 2022-08-17T12:29:54.0158978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq2yo4gx 2022-08-17T12:29:54.0160066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq2yo4gx/_remote_module_non_scriptable.py 2022-08-17T12:29:54.2333874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:54.2417904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:54.2800602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:54.2823173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:54.5597891Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T12:29:54.5598103Z 2022-08-17T12:29:54.5598422Z ---------------------------------------------------------------------- 2022-08-17T12:29:54.5598696Z Ran 1 test in 2.076s 2022-08-17T12:29:54.5598803Z 2022-08-17T12:29:54.5598881Z OK (skipped=1) 2022-08-17T12:29:54.5598996Z 2022-08-17T12:29:54.5599085Z Generating XML reports... 2022-08-17T12:29:54.5637024Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122952.xml 2022-08-17T12:29:55.8409848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:55.8410250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:55.8412723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:55.8413124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:55.9926149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hy92in3 2022-08-17T12:29:55.9929789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hy92in3/_remote_module_non_scriptable.py 2022-08-17T12:29:56.2500105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:29:56.2510745Z 2022-08-17T12:29:56.2510852Z Running tests... 2022-08-17T12:29:56.2511291Z ---------------------------------------------------------------------- 2022-08-17T12:29:56.5116429Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26283 2022-08-17T12:29:56.5146127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26284 2022-08-17T12:29:56.5174885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26285 2022-08-17T12:29:56.5206543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26286 2022-08-17T12:29:57.5315748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:57.5316395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:57.5319061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:57.5319708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:57.5409546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:57.5410200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:57.5414231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:57.5414926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:57.5469052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:57.5469687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:57.5473544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:57.5474190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:57.5640632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:57.5641284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:57.5644505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:57.5645455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:57.6940944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn01wxddl 2022-08-17T12:29:57.6941703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn01wxddl/_remote_module_non_scriptable.py 2022-08-17T12:29:57.7028944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccfgt7yi 2022-08-17T12:29:57.7030093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccfgt7yi/_remote_module_non_scriptable.py 2022-08-17T12:29:57.7235414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo99ocz51 2022-08-17T12:29:57.7236625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo99ocz51/_remote_module_non_scriptable.py 2022-08-17T12:29:57.7271836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppln8vu_b 2022-08-17T12:29:57.7273252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppln8vu_b/_remote_module_non_scriptable.py 2022-08-17T12:29:57.9567902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:29:57.9646273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:29:57.9901266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:29:57.9931939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:29:58.3256203Z skip: Need at least 1 CUDA device (2.074s) 2022-08-17T12:29:58.3256504Z 2022-08-17T12:29:58.3256962Z ---------------------------------------------------------------------- 2022-08-17T12:29:58.3257375Z Ran 1 test in 2.074s 2022-08-17T12:29:58.3257568Z 2022-08-17T12:29:58.3257691Z OK (skipped=1) 2022-08-17T12:29:58.3257866Z 2022-08-17T12:29:58.3257986Z Generating XML reports... 2022-08-17T12:29:58.3294425Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122956.xml 2022-08-17T12:29:59.6095038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:29:59.6095514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:29:59.6097386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:29:59.6097765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:29:59.7609594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyz6r8ih 2022-08-17T12:29:59.7612773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyz6r8ih/_remote_module_non_scriptable.py 2022-08-17T12:30:00.0175994Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:00.0187679Z 2022-08-17T12:30:00.0187831Z Running tests... 2022-08-17T12:30:00.0188908Z ---------------------------------------------------------------------- 2022-08-17T12:30:00.2842015Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26338 2022-08-17T12:30:00.2871583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26339 2022-08-17T12:30:00.2900717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26340 2022-08-17T12:30:00.2933113Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26341 2022-08-17T12:30:01.3719618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:01.3720479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:01.3765621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:01.3766928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:01.3791446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:01.3792118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:01.3795921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:01.3796545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:01.3812422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:01.3813064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:01.3816650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:01.3817318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:01.4016248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:01.4017200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:01.4020015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:01.4020696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:01.5335336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96nch4ma 2022-08-17T12:30:01.5338429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96nch4ma/_remote_module_non_scriptable.py 2022-08-17T12:30:01.5430214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8pl6cb1 2022-08-17T12:30:01.5431049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8pl6cb1/_remote_module_non_scriptable.py 2022-08-17T12:30:01.5431791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr0ade8w 2022-08-17T12:30:01.5432569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr0ade8w/_remote_module_non_scriptable.py 2022-08-17T12:30:01.5611464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx72dgzn 2022-08-17T12:30:01.5612180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx72dgzn/_remote_module_non_scriptable.py 2022-08-17T12:30:01.8012932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:01.8079680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:01.8249292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:01.8286480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:02.1983843Z skip: Need at least 2 CUDA devices (2.179s) 2022-08-17T12:30:02.1984066Z 2022-08-17T12:30:02.1984539Z ---------------------------------------------------------------------- 2022-08-17T12:30:02.1984833Z Ran 1 test in 2.180s 2022-08-17T12:30:02.1984955Z 2022-08-17T12:30:02.1985034Z OK (skipped=1) 2022-08-17T12:30:02.1985194Z 2022-08-17T12:30:02.1985325Z Generating XML reports... 2022-08-17T12:30:02.2021401Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123000.xml 2022-08-17T12:30:03.5177401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:03.5177795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:03.5180140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:03.5180810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:03.6694640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0mkzjug 2022-08-17T12:30:03.6698479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0mkzjug/_remote_module_non_scriptable.py 2022-08-17T12:30:03.9287705Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:03.9298166Z 2022-08-17T12:30:03.9298310Z Running tests... 2022-08-17T12:30:03.9298924Z ---------------------------------------------------------------------- 2022-08-17T12:30:04.1919519Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26393 2022-08-17T12:30:04.1947730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26394 2022-08-17T12:30:04.1976473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26395 2022-08-17T12:30:04.2006680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26396 2022-08-17T12:30:05.2846028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:05.2846992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:05.2849748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:05.2850411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:05.3240870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:05.3244526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:05.3245445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:05.3246305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:05.3295343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:05.3296053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:05.3299508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:05.3300159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:05.3462643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:05.3463286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:05.3466188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:05.3466824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:05.4407969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ykl81og 2022-08-17T12:30:05.4409030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ykl81og/_remote_module_non_scriptable.py 2022-08-17T12:30:05.4832327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0o9rw5j 2022-08-17T12:30:05.4833124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0o9rw5j/_remote_module_non_scriptable.py 2022-08-17T12:30:05.4877530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qddbsml 2022-08-17T12:30:05.4878795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qddbsml/_remote_module_non_scriptable.py 2022-08-17T12:30:05.5039745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfl9wg8k 2022-08-17T12:30:05.5040556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfl9wg8k/_remote_module_non_scriptable.py 2022-08-17T12:30:05.6983741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:05.7425297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:05.7473582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:05.7664708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:06.1059702Z skip: Need at least 2 CUDA devices (2.176s) 2022-08-17T12:30:06.1060039Z 2022-08-17T12:30:06.1060552Z ---------------------------------------------------------------------- 2022-08-17T12:30:06.1060801Z Ran 1 test in 2.176s 2022-08-17T12:30:06.1060922Z 2022-08-17T12:30:06.1060999Z OK (skipped=1) 2022-08-17T12:30:06.1061112Z 2022-08-17T12:30:06.1061202Z Generating XML reports... 2022-08-17T12:30:06.1099160Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123003.xml 2022-08-17T12:30:07.3960957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:07.3961648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:07.3964854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:07.3965495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:07.5456178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9vq5zeg 2022-08-17T12:30:07.5459811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9vq5zeg/_remote_module_non_scriptable.py 2022-08-17T12:30:07.8045441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:07.8057052Z 2022-08-17T12:30:07.8057361Z Running tests... 2022-08-17T12:30:07.8057998Z ---------------------------------------------------------------------- 2022-08-17T12:30:08.0739600Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26448 2022-08-17T12:30:08.0770199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26449 2022-08-17T12:30:08.0803508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26450 2022-08-17T12:30:08.0832808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26451 2022-08-17T12:30:09.1375889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:09.1376544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:09.1379674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:09.1380316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:09.1485806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:09.1486670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:09.1489553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:09.1490200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:09.1693490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:09.1694324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:09.1695707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:09.1696580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:09.1849321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:09.1849984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:09.1853037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:09.1853581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:09.2965446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zb1bphg 2022-08-17T12:30:09.2966740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zb1bphg/_remote_module_non_scriptable.py 2022-08-17T12:30:09.3078186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59nrwyv_ 2022-08-17T12:30:09.3079472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59nrwyv_/_remote_module_non_scriptable.py 2022-08-17T12:30:09.3294232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusz6gmlc 2022-08-17T12:30:09.3295246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusz6gmlc/_remote_module_non_scriptable.py 2022-08-17T12:30:09.3450678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe70772qi 2022-08-17T12:30:09.3452008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe70772qi/_remote_module_non_scriptable.py 2022-08-17T12:30:09.5578823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:09.5785970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:09.5913298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:09.6224772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:09.9885275Z skip: Need at least 2 CUDA devices (2.182s) 2022-08-17T12:30:09.9885588Z 2022-08-17T12:30:09.9886284Z ---------------------------------------------------------------------- 2022-08-17T12:30:09.9886750Z Ran 1 test in 2.183s 2022-08-17T12:30:09.9886962Z 2022-08-17T12:30:09.9887077Z OK (skipped=1) 2022-08-17T12:30:09.9887261Z 2022-08-17T12:30:09.9887354Z Generating XML reports... 2022-08-17T12:30:09.9922387Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123007.xml 2022-08-17T12:30:11.2902094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:11.2902790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:11.2905727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:11.2925535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:11.4425279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz447vfb2 2022-08-17T12:30:11.4429233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz447vfb2/_remote_module_non_scriptable.py 2022-08-17T12:30:11.7016375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:11.7027981Z 2022-08-17T12:30:11.7028296Z Running tests... 2022-08-17T12:30:11.7028991Z ---------------------------------------------------------------------- 2022-08-17T12:30:11.9688738Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26503 2022-08-17T12:30:11.9717800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26504 2022-08-17T12:30:11.9747199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26505 2022-08-17T12:30:11.9779398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26506 2022-08-17T12:30:13.0032999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:13.0033653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:13.0036703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:13.0037404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:13.0447290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:13.0447951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:13.0451185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:13.0451839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:13.1114974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:13.1115605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:13.1118845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:13.1119494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:13.1209440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:13.1210036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:13.1214629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:13.1215291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:13.1603572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_z6_p09 2022-08-17T12:30:13.1604422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_z6_p09/_remote_module_non_scriptable.py 2022-08-17T12:30:13.2026329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl59dnaum 2022-08-17T12:30:13.2028079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl59dnaum/_remote_module_non_scriptable.py 2022-08-17T12:30:13.2682921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyiqwqxm 2022-08-17T12:30:13.2683744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyiqwqxm/_remote_module_non_scriptable.py 2022-08-17T12:30:13.2790461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeinh7xmd 2022-08-17T12:30:13.2791336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeinh7xmd/_remote_module_non_scriptable.py 2022-08-17T12:30:13.4241259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:13.4868143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:13.5263884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:13.5752339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:13.8830564Z skip: Need at least 1 CUDA device (2.180s) 2022-08-17T12:30:13.8831010Z 2022-08-17T12:30:13.8831464Z ---------------------------------------------------------------------- 2022-08-17T12:30:13.8831709Z Ran 1 test in 2.180s 2022-08-17T12:30:13.8831838Z 2022-08-17T12:30:13.8831915Z OK (skipped=1) 2022-08-17T12:30:13.8832027Z 2022-08-17T12:30:13.8832121Z Generating XML reports... 2022-08-17T12:30:13.8869108Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123011.xml 2022-08-17T12:30:15.1972933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:15.1973336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:15.1975635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:15.1976090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:15.3523871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u8wfpz6 2022-08-17T12:30:15.3527290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u8wfpz6/_remote_module_non_scriptable.py 2022-08-17T12:30:15.6116754Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:15.6127706Z 2022-08-17T12:30:15.6127838Z Running tests... 2022-08-17T12:30:15.6128769Z ---------------------------------------------------------------------- 2022-08-17T12:30:15.8740571Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26558 2022-08-17T12:30:15.8768987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26559 2022-08-17T12:30:15.8798242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26560 2022-08-17T12:30:15.8830379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26561 2022-08-17T12:30:16.8904226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:16.8904892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:16.8906442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:16.8907083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:16.9006587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:16.9007230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:16.9010472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:16.9011125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:16.9111065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:16.9111762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:16.9115430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:16.9116064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:16.9248964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:16.9249617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:16.9252548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:16.9253188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:17.0497772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvazbpj0 2022-08-17T12:30:17.0498666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvazbpj0/_remote_module_non_scriptable.py 2022-08-17T12:30:17.0602485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjeqywoe 2022-08-17T12:30:17.0603772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjeqywoe/_remote_module_non_scriptable.py 2022-08-17T12:30:17.0725965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxzptz29 2022-08-17T12:30:17.0726720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxzptz29/_remote_module_non_scriptable.py 2022-08-17T12:30:17.0847678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7h0cjmg 2022-08-17T12:30:17.0848556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7h0cjmg/_remote_module_non_scriptable.py 2022-08-17T12:30:17.3142158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:17.3263132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:17.3401543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:17.3507193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:17.6881420Z skip: Need at least 1 CUDA device (2.075s) 2022-08-17T12:30:17.6881796Z 2022-08-17T12:30:17.6882237Z ---------------------------------------------------------------------- 2022-08-17T12:30:17.6882505Z Ran 1 test in 2.075s 2022-08-17T12:30:17.6882626Z 2022-08-17T12:30:17.6882703Z OK (skipped=1) 2022-08-17T12:30:17.6882804Z 2022-08-17T12:30:17.6883138Z Generating XML reports... 2022-08-17T12:30:17.6918791Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123015.xml 2022-08-17T12:30:18.9837511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:18.9838135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:18.9840367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:18.9840970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:19.1355481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97rhe_5_ 2022-08-17T12:30:19.1358487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97rhe_5_/_remote_module_non_scriptable.py 2022-08-17T12:30:19.3918320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:19.3929040Z 2022-08-17T12:30:19.3929563Z Running tests... 2022-08-17T12:30:19.3930006Z ---------------------------------------------------------------------- 2022-08-17T12:30:19.6618752Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26613 2022-08-17T12:30:19.6645397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26614 2022-08-17T12:30:19.6674196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26615 2022-08-17T12:30:19.6706115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26616 2022-08-17T12:30:20.7018216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:20.7018915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:20.7021104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:20.7021767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:20.7524077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:20.7524741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:20.7528281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:20.7528918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:20.7853036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:20.7853650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:20.7855975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:20.7856602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:20.7938861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:20.7939509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:20.7942285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:20.7942913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:20.8614588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zvlz7az 2022-08-17T12:30:20.8615506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zvlz7az/_remote_module_non_scriptable.py 2022-08-17T12:30:20.9120917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhm56_cg 2022-08-17T12:30:20.9122161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhm56_cg/_remote_module_non_scriptable.py 2022-08-17T12:30:20.9444408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibppajyy 2022-08-17T12:30:20.9445201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibppajyy/_remote_module_non_scriptable.py 2022-08-17T12:30:20.9536586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qvt5fcv 2022-08-17T12:30:20.9537313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qvt5fcv/_remote_module_non_scriptable.py 2022-08-17T12:30:21.1247213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:21.1833188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:21.2069521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:21.2183968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:21.5759076Z skip: Need at least 1 CUDA device (2.183s) 2022-08-17T12:30:21.5759258Z 2022-08-17T12:30:21.5759577Z ---------------------------------------------------------------------- 2022-08-17T12:30:21.5759928Z Ran 1 test in 2.183s 2022-08-17T12:30:21.5760036Z 2022-08-17T12:30:21.5760111Z OK (skipped=1) 2022-08-17T12:30:21.5760225Z 2022-08-17T12:30:21.5760316Z Generating XML reports... 2022-08-17T12:30:21.5796806Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123019.xml 2022-08-17T12:30:22.8910647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:22.8911047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:22.8913253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:22.8913727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:23.0459087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk76tq5wf 2022-08-17T12:30:23.0462560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk76tq5wf/_remote_module_non_scriptable.py 2022-08-17T12:30:23.3085507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:23.3096729Z 2022-08-17T12:30:23.3096870Z Running tests... 2022-08-17T12:30:23.3097320Z ---------------------------------------------------------------------- 2022-08-17T12:30:23.5754454Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26668 2022-08-17T12:30:23.5784619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26669 2022-08-17T12:30:23.5813811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26670 2022-08-17T12:30:23.5845264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26671 2022-08-17T12:30:24.5936280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:24.5936902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:24.5939391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:24.5940002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:24.6167277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:24.6167943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:24.6171256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:24.6171930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:24.6478625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:24.6479432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:24.6483223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:24.6483878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:24.6760335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:24.6761007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:24.6764240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:24.6764806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:24.7578084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln48hko3 2022-08-17T12:30:24.7578976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln48hko3/_remote_module_non_scriptable.py 2022-08-17T12:30:24.7781609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp448az1gc 2022-08-17T12:30:24.7782394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp448az1gc/_remote_module_non_scriptable.py 2022-08-17T12:30:24.8126584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fxvs4k1 2022-08-17T12:30:24.8127442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fxvs4k1/_remote_module_non_scriptable.py 2022-08-17T12:30:24.8452712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9lj75mn 2022-08-17T12:30:24.8453559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9lj75mn/_remote_module_non_scriptable.py 2022-08-17T12:30:25.0218121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:25.0429393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:25.0811519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:25.1115140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:25.3896219Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T12:30:25.3896956Z 2022-08-17T12:30:25.3897346Z ---------------------------------------------------------------------- 2022-08-17T12:30:25.3897612Z Ran 1 test in 2.080s 2022-08-17T12:30:25.3897733Z 2022-08-17T12:30:25.3897809Z OK (skipped=1) 2022-08-17T12:30:25.3897912Z 2022-08-17T12:30:25.3898004Z Generating XML reports... 2022-08-17T12:30:25.3933755Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123023.xml 2022-08-17T12:30:26.6915418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:26.6915820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:26.6918279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:26.6918657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:26.8436974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_531099b 2022-08-17T12:30:26.8439814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_531099b/_remote_module_non_scriptable.py 2022-08-17T12:30:27.1036661Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:27.1047623Z 2022-08-17T12:30:27.1047781Z Running tests... 2022-08-17T12:30:27.1048170Z ---------------------------------------------------------------------- 2022-08-17T12:30:27.3676051Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26723 2022-08-17T12:30:27.3704418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26724 2022-08-17T12:30:27.3732529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26725 2022-08-17T12:30:27.3763238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26726 2022-08-17T12:30:28.3641541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:28.3642185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:28.3644796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:28.3645469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:28.4054428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:28.4055106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:28.4058105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:28.4058678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:28.4108441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:28.4109132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:28.4113044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:28.4113684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:28.4206861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:28.4207518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:28.4210619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:28.4211282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:28.5237562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qxayl2k 2022-08-17T12:30:28.5238680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qxayl2k/_remote_module_non_scriptable.py 2022-08-17T12:30:28.5636995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu27m4sx2 2022-08-17T12:30:28.5637852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu27m4sx2/_remote_module_non_scriptable.py 2022-08-17T12:30:28.5710712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepanjf5o 2022-08-17T12:30:28.5711656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepanjf5o/_remote_module_non_scriptable.py 2022-08-17T12:30:28.5796959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeodgp0de 2022-08-17T12:30:28.5798157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeodgp0de/_remote_module_non_scriptable.py 2022-08-17T12:30:28.7893600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:28.8230521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:28.8344743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:28.8507129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:29.1814930Z skip: Need at least 1 CUDA device (2.076s) 2022-08-17T12:30:29.1815137Z 2022-08-17T12:30:29.1815547Z ---------------------------------------------------------------------- 2022-08-17T12:30:29.1815827Z Ran 1 test in 2.077s 2022-08-17T12:30:29.1815936Z 2022-08-17T12:30:29.1816012Z OK (skipped=1) 2022-08-17T12:30:29.1816160Z 2022-08-17T12:30:29.1816296Z Generating XML reports... 2022-08-17T12:30:29.1852027Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123027.xml 2022-08-17T12:30:30.4896308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:30.4896730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:30.4898876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:30.4899328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:30.6418745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvp14cki 2022-08-17T12:30:30.6421579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvp14cki/_remote_module_non_scriptable.py 2022-08-17T12:30:30.9008235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:30.9020223Z 2022-08-17T12:30:30.9020363Z Running tests... 2022-08-17T12:30:30.9020803Z ---------------------------------------------------------------------- 2022-08-17T12:30:31.1680634Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26778 2022-08-17T12:30:31.1709249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26779 2022-08-17T12:30:31.1738023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26780 2022-08-17T12:30:31.1769199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26781 2022-08-17T12:30:32.1745713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:32.1746381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:32.1748782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:32.1749434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:32.1751351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:32.1752298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:32.1757245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:32.1757897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:32.2089810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:32.2090458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:32.2093247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:32.2093933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:32.2385534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:32.2386233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:32.2389069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:32.2390020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:32.3334639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rnjayby 2022-08-17T12:30:32.3335438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rnjayby/_remote_module_non_scriptable.py 2022-08-17T12:30:32.3339576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t9xg8mp 2022-08-17T12:30:32.3342087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t9xg8mp/_remote_module_non_scriptable.py 2022-08-17T12:30:32.3670545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9h27jak 2022-08-17T12:30:32.3671349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9h27jak/_remote_module_non_scriptable.py 2022-08-17T12:30:32.3981605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjoob1kl2 2022-08-17T12:30:32.3982444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjoob1kl2/_remote_module_non_scriptable.py 2022-08-17T12:30:32.5912488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:32.5924604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:32.6278416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:32.6636404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:32.9820714Z skip: Need at least 1 CUDA device (2.080s) 2022-08-17T12:30:32.9821018Z 2022-08-17T12:30:32.9821342Z ---------------------------------------------------------------------- 2022-08-17T12:30:32.9821621Z Ran 1 test in 2.080s 2022-08-17T12:30:32.9821742Z 2022-08-17T12:30:32.9821804Z OK (skipped=1) 2022-08-17T12:30:32.9821917Z 2022-08-17T12:30:32.9822008Z Generating XML reports... 2022-08-17T12:30:32.9857641Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123030.xml 2022-08-17T12:30:34.2774069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:34.2774471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:34.2776901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:34.2777282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:34.4282518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8l9kxmx 2022-08-17T12:30:34.4286370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8l9kxmx/_remote_module_non_scriptable.py 2022-08-17T12:30:34.6869867Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:34.6880558Z 2022-08-17T12:30:34.6880669Z Running tests... 2022-08-17T12:30:34.6881158Z ---------------------------------------------------------------------- 2022-08-17T12:30:34.9497687Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26833 2022-08-17T12:30:34.9526662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26834 2022-08-17T12:30:34.9555041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26835 2022-08-17T12:30:34.9587145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26836 2022-08-17T12:30:35.9500323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:35.9500997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:35.9503586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:35.9504238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:35.9664663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:35.9665270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:35.9668787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:35.9669423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:35.9760461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:35.9761149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:35.9764428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:35.9765050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:35.9989595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:35.9990285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:35.9993849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:35.9994473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:36.1089751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkud8unbb 2022-08-17T12:30:36.1090622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkud8unbb/_remote_module_non_scriptable.py 2022-08-17T12:30:36.1245071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4amsi2o 2022-08-17T12:30:36.1247361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4amsi2o/_remote_module_non_scriptable.py 2022-08-17T12:30:36.1365018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz6ld8an 2022-08-17T12:30:36.1367622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz6ld8an/_remote_module_non_scriptable.py 2022-08-17T12:30:36.1591431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekehgn94 2022-08-17T12:30:36.1592220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekehgn94/_remote_module_non_scriptable.py 2022-08-17T12:30:36.3840893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:36.3918462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:36.4088057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:36.4240594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:36.7637231Z skip: Need at least 1 CUDA device (2.075s) 2022-08-17T12:30:36.7637571Z 2022-08-17T12:30:36.7638011Z ---------------------------------------------------------------------- 2022-08-17T12:30:36.7638272Z Ran 1 test in 2.076s 2022-08-17T12:30:36.7638393Z 2022-08-17T12:30:36.7638469Z OK (skipped=1) 2022-08-17T12:30:36.7638582Z 2022-08-17T12:30:36.7638658Z Generating XML reports... 2022-08-17T12:30:36.7677231Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123034.xml 2022-08-17T12:30:38.0469577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:38.0470296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:38.0473565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:38.0474442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:38.1958811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf8l5e7o 2022-08-17T12:30:38.1962271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf8l5e7o/_remote_module_non_scriptable.py 2022-08-17T12:30:38.4517367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:38.4529472Z 2022-08-17T12:30:38.4529767Z Running tests... 2022-08-17T12:30:38.4530209Z ---------------------------------------------------------------------- 2022-08-17T12:30:38.7103085Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26888 2022-08-17T12:30:38.7133317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26889 2022-08-17T12:30:38.7161891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26890 2022-08-17T12:30:38.7194656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26891 2022-08-17T12:30:39.7467557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:39.7468235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:39.7471176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:39.7471826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:39.7523742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:39.7524410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:39.7528369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:39.7529052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:39.7560053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:39.7560681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:39.7564482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:39.7565150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:39.7714616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:39.7715585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:39.7718485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:39.7719146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:39.9075877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz3rfnsk 2022-08-17T12:30:39.9077847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz3rfnsk/_remote_module_non_scriptable.py 2022-08-17T12:30:39.9114165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxmnfpxe 2022-08-17T12:30:39.9115370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxmnfpxe/_remote_module_non_scriptable.py 2022-08-17T12:30:39.9157582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu89qmucy 2022-08-17T12:30:39.9159354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu89qmucy/_remote_module_non_scriptable.py 2022-08-17T12:30:39.9294474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8372b1n 2022-08-17T12:30:39.9295678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8372b1n/_remote_module_non_scriptable.py 2022-08-17T12:30:40.1699813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:40.1700304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:40.1802179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:40.1897397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:40.5244753Z skip: Need at least 1 CUDA device (2.071s) 2022-08-17T12:30:40.5244982Z 2022-08-17T12:30:40.5245382Z ---------------------------------------------------------------------- 2022-08-17T12:30:40.5245668Z Ran 1 test in 2.071s 2022-08-17T12:30:40.5245828Z 2022-08-17T12:30:40.5246104Z OK (skipped=1) 2022-08-17T12:30:40.5246245Z 2022-08-17T12:30:40.5246322Z Generating XML reports... 2022-08-17T12:30:40.5282946Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123038.xml 2022-08-17T12:30:41.8200846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:41.8201222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:41.8203563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:41.8204096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:41.9724006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnxi68hl 2022-08-17T12:30:41.9727069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnxi68hl/_remote_module_non_scriptable.py 2022-08-17T12:30:42.2312931Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:42.2323346Z 2022-08-17T12:30:42.2323443Z Running tests... 2022-08-17T12:30:42.2324540Z ---------------------------------------------------------------------- 2022-08-17T12:30:42.4981705Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26943 2022-08-17T12:30:42.5012735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26944 2022-08-17T12:30:42.5042444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26945 2022-08-17T12:30:42.5075335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26946 2022-08-17T12:30:43.4866060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:43.4867006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:43.4869962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:43.4870552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:43.5326475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:43.5327153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:43.5330089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:43.5330751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:43.5515294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:43.5515928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:43.5518781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:43.5519682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:43.5671829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:43.5672523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:43.5676428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:43.5677121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:43.6444726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebcvu5a5 2022-08-17T12:30:43.6445471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebcvu5a5/_remote_module_non_scriptable.py 2022-08-17T12:30:43.6902248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputsakaf2 2022-08-17T12:30:43.6903251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputsakaf2/_remote_module_non_scriptable.py 2022-08-17T12:30:43.7095387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad7xtak9 2022-08-17T12:30:43.7096496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad7xtak9/_remote_module_non_scriptable.py 2022-08-17T12:30:43.7257882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sn1ovui 2022-08-17T12:30:43.7258833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sn1ovui/_remote_module_non_scriptable.py 2022-08-17T12:30:43.9033298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:43.9466361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:43.9697497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:43.9896681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:44.3125238Z skip: Need at least 1 CUDA device (2.080s) 2022-08-17T12:30:44.3125552Z 2022-08-17T12:30:44.3126289Z ---------------------------------------------------------------------- 2022-08-17T12:30:44.3126561Z Ran 1 test in 2.080s 2022-08-17T12:30:44.3126668Z 2022-08-17T12:30:44.3126742Z OK (skipped=1) 2022-08-17T12:30:44.3126859Z 2022-08-17T12:30:44.3126949Z Generating XML reports... 2022-08-17T12:30:44.3162874Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123042.xml 2022-08-17T12:30:45.6297092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:45.6297764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:45.6299873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:45.6300250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:45.7817005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsn7b8yyb 2022-08-17T12:30:45.7819555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsn7b8yyb/_remote_module_non_scriptable.py 2022-08-17T12:30:46.0400471Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:46.0411247Z 2022-08-17T12:30:46.0411617Z Running tests... 2022-08-17T12:30:46.0412023Z ---------------------------------------------------------------------- 2022-08-17T12:30:46.3072823Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26998 2022-08-17T12:30:46.3102361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26999 2022-08-17T12:30:46.3132529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27000 2022-08-17T12:30:46.3164683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27001 2022-08-17T12:30:47.3368866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:47.3369519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:47.3372093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:47.3372763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:47.3472708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:47.3473385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:47.3476919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:47.3477574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:47.3915361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:47.3915926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:47.3919283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:47.3919942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:47.4057143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:47.4057709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:47.4061255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:47.4061915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:47.4937201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputjxyayj 2022-08-17T12:30:47.4937958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputjxyayj/_remote_module_non_scriptable.py 2022-08-17T12:30:47.5046879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4px5es_s 2022-08-17T12:30:47.5048075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4px5es_s/_remote_module_non_scriptable.py 2022-08-17T12:30:47.5500904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4luselm_ 2022-08-17T12:30:47.5501774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4luselm_/_remote_module_non_scriptable.py 2022-08-17T12:30:47.5631350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm82t9sq8 2022-08-17T12:30:47.5632520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm82t9sq8/_remote_module_non_scriptable.py 2022-08-17T12:30:47.7536439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:47.7666874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:47.8164265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:47.8505824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:48.1214351Z skip: Need at least 1 CUDA device (2.080s) 2022-08-17T12:30:48.1214654Z 2022-08-17T12:30:48.1214974Z ---------------------------------------------------------------------- 2022-08-17T12:30:48.1215251Z Ran 1 test in 2.080s 2022-08-17T12:30:48.1215382Z 2022-08-17T12:30:48.1215458Z OK (skipped=1) 2022-08-17T12:30:48.1215574Z 2022-08-17T12:30:48.1215664Z Generating XML reports... 2022-08-17T12:30:48.1250670Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123046.xml 2022-08-17T12:30:49.4408452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:49.4408882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:49.4411314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:49.4411693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:49.5945296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqvnghhr 2022-08-17T12:30:49.5948915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqvnghhr/_remote_module_non_scriptable.py 2022-08-17T12:30:49.8569974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:49.8581596Z 2022-08-17T12:30:49.8581889Z Running tests... 2022-08-17T12:30:49.8582599Z ---------------------------------------------------------------------- 2022-08-17T12:30:50.1239245Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27053 2022-08-17T12:30:50.1267087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27054 2022-08-17T12:30:50.1296151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27055 2022-08-17T12:30:50.1327135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27056 2022-08-17T12:30:51.1323375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:51.1324052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:51.1327373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:51.1328055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:51.1484193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:51.1484877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:51.1488058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:51.1488743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:51.1657694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:51.1658572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:51.1661462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:51.1662244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:51.1693219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:51.1693839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:51.1698601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:51.1699270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:51.2920494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggq9a3zl 2022-08-17T12:30:51.2921624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggq9a3zl/_remote_module_non_scriptable.py 2022-08-17T12:30:51.3089377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvryxpm59 2022-08-17T12:30:51.3090233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvryxpm59/_remote_module_non_scriptable.py 2022-08-17T12:30:51.3253055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy6c10qu 2022-08-17T12:30:51.3253872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy6c10qu/_remote_module_non_scriptable.py 2022-08-17T12:30:51.3293730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4u9p6l8 2022-08-17T12:30:51.3295552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4u9p6l8/_remote_module_non_scriptable.py 2022-08-17T12:30:51.5525707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:51.5725257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:51.5870865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:51.5921977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:51.9378340Z skip: Need at least 1 CUDA device (2.079s) 2022-08-17T12:30:51.9378651Z 2022-08-17T12:30:51.9379209Z ---------------------------------------------------------------------- 2022-08-17T12:30:51.9379579Z Ran 1 test in 2.080s 2022-08-17T12:30:51.9379686Z 2022-08-17T12:30:51.9379768Z OK (skipped=1) 2022-08-17T12:30:51.9379881Z 2022-08-17T12:30:51.9379973Z Generating XML reports... 2022-08-17T12:30:51.9416644Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123049.xml 2022-08-17T12:30:53.2528585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:53.2528977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:53.2531771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:53.2532260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:53.4070096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqunfyu5 2022-08-17T12:30:53.4073346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqunfyu5/_remote_module_non_scriptable.py 2022-08-17T12:30:53.6656108Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:53.6667204Z 2022-08-17T12:30:53.6667306Z Running tests... 2022-08-17T12:30:53.6668265Z ---------------------------------------------------------------------- 2022-08-17T12:30:53.9325142Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27108 2022-08-17T12:30:53.9353734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27109 2022-08-17T12:30:53.9382876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27110 2022-08-17T12:30:53.9411812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27111 2022-08-17T12:30:55.0027885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:55.0028588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:55.0030931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:55.0031607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:55.0555470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:55.0556160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:55.0559742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:55.0560375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:55.0790554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:55.0791238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:55.0793365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:55.0794038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:55.0832374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:55.0833031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:55.0836634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:55.0837282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:55.1606500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9b09or3 2022-08-17T12:30:55.1607378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9b09or3/_remote_module_non_scriptable.py 2022-08-17T12:30:55.2137233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67u1n178 2022-08-17T12:30:55.2138049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67u1n178/_remote_module_non_scriptable.py 2022-08-17T12:30:55.2377352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqy18onl 2022-08-17T12:30:55.2378585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqy18onl/_remote_module_non_scriptable.py 2022-08-17T12:30:55.2427508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf1neh0y 2022-08-17T12:30:55.2428891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf1neh0y/_remote_module_non_scriptable.py 2022-08-17T12:30:55.4236638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:55.4734927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:55.4997436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:55.5066237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:55.8463777Z skip: Need at least 2 CUDA devices (2.179s) 2022-08-17T12:30:55.8464101Z 2022-08-17T12:30:55.8464484Z ---------------------------------------------------------------------- 2022-08-17T12:30:55.8464749Z Ran 1 test in 2.180s 2022-08-17T12:30:55.8465198Z 2022-08-17T12:30:55.8465711Z OK (skipped=1) 2022-08-17T12:30:55.8465866Z 2022-08-17T12:30:55.8465959Z Generating XML reports... 2022-08-17T12:30:55.8501687Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123053.xml 2022-08-17T12:30:57.1670147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:57.1672595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:57.1673069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:57.1673439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:57.3197166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp913om9zy 2022-08-17T12:30:57.3200964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp913om9zy/_remote_module_non_scriptable.py 2022-08-17T12:30:57.5782482Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:30:57.5793962Z 2022-08-17T12:30:57.5794282Z Running tests... 2022-08-17T12:30:57.5794946Z ---------------------------------------------------------------------- 2022-08-17T12:30:57.8441129Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27163 2022-08-17T12:30:57.8469423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27164 2022-08-17T12:30:57.8498691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27165 2022-08-17T12:30:57.8530786Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27166 2022-08-17T12:30:58.9062370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:58.9063073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:58.9065270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:58.9065943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:58.9662730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:58.9663372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:58.9666674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:58.9667236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:58.9993988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:58.9994638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:58.9997883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:58.9998554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:59.0124764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:30:59.0125468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:30:59.0129282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:30:59.0129897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:30:59.0648001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9me600w 2022-08-17T12:30:59.0648777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9me600w/_remote_module_non_scriptable.py 2022-08-17T12:30:59.1251957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgf22klj 2022-08-17T12:30:59.1253128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgf22klj/_remote_module_non_scriptable.py 2022-08-17T12:30:59.1724205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzflaepj 2022-08-17T12:30:59.1725341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzflaepj/_remote_module_non_scriptable.py 2022-08-17T12:30:59.1746336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8keosqt 2022-08-17T12:30:59.1747931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8keosqt/_remote_module_non_scriptable.py 2022-08-17T12:30:59.3300994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:30:59.3857165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:30:59.4341185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:30:59.4505609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:30:59.7581685Z skip: Need at least 2 CUDA devices (2.178s) 2022-08-17T12:30:59.7581934Z 2022-08-17T12:30:59.7582560Z ---------------------------------------------------------------------- 2022-08-17T12:30:59.7582833Z Ran 1 test in 2.179s 2022-08-17T12:30:59.7582960Z 2022-08-17T12:30:59.7583037Z OK (skipped=1) 2022-08-17T12:30:59.7583174Z 2022-08-17T12:30:59.7583313Z Generating XML reports... 2022-08-17T12:30:59.7618623Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123057.xml 2022-08-17T12:31:01.0526884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:01.0527284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:01.0530201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:01.0530577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:01.2065005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlvmoi9x 2022-08-17T12:31:01.2068673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlvmoi9x/_remote_module_non_scriptable.py 2022-08-17T12:31:01.4716412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:01.4727649Z 2022-08-17T12:31:01.4728124Z Running tests... 2022-08-17T12:31:01.4728555Z ---------------------------------------------------------------------- 2022-08-17T12:31:01.7400321Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27218 2022-08-17T12:31:01.7429733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27219 2022-08-17T12:31:01.7457915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27220 2022-08-17T12:31:01.7489660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27221 2022-08-17T12:31:02.8606532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:02.8607143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:02.8610133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:02.8610792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:02.8646262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:02.8646968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:02.8651242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:02.8651881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:02.9102258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:02.9102938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:02.9104948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:02.9105556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:02.9467130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:02.9467822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:02.9470925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:02.9471594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:03.0204442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf500ko2d 2022-08-17T12:31:03.0205501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf500ko2d/_remote_module_non_scriptable.py 2022-08-17T12:31:03.0248830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnjc4wyy 2022-08-17T12:31:03.0249869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnjc4wyy/_remote_module_non_scriptable.py 2022-08-17T12:31:03.0718898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9155222 2022-08-17T12:31:03.0719749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9155222/_remote_module_non_scriptable.py 2022-08-17T12:31:03.1038657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizgq9pqt 2022-08-17T12:31:03.1039499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizgq9pqt/_remote_module_non_scriptable.py 2022-08-17T12:31:03.2875164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:03.2920572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:03.3368715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:03.4081779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:03.7542927Z skip: Need at least 2 CUDA devices (2.281s) 2022-08-17T12:31:03.7543236Z 2022-08-17T12:31:03.7543619Z ---------------------------------------------------------------------- 2022-08-17T12:31:03.7543885Z Ran 1 test in 2.281s 2022-08-17T12:31:03.7544006Z 2022-08-17T12:31:03.7544068Z OK (skipped=1) 2022-08-17T12:31:03.7544185Z 2022-08-17T12:31:03.7544299Z Generating XML reports... 2022-08-17T12:31:03.7579781Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123101.xml 2022-08-17T12:31:05.0591957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:05.0592360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:05.0594847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:05.0595231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:05.2123275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeld38tfi 2022-08-17T12:31:05.2127460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeld38tfi/_remote_module_non_scriptable.py 2022-08-17T12:31:05.4727678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:05.4738916Z 2022-08-17T12:31:05.4739377Z Running tests... 2022-08-17T12:31:05.4739780Z ---------------------------------------------------------------------- 2022-08-17T12:31:05.7347727Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27273 2022-08-17T12:31:05.7374584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27274 2022-08-17T12:31:05.7403107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27275 2022-08-17T12:31:05.7435680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27276 2022-08-17T12:31:06.7155115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:06.7155812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:06.7157297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:06.7157906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:06.7159000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:06.7159634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:06.7160482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:06.7161061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:06.7163403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:06.7164066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:06.7165051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:06.7165714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:06.7814825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:06.7815425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:06.7818278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:06.7818888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:06.8741028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nds31cc 2022-08-17T12:31:06.8741831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nds31cc/_remote_module_non_scriptable.py 2022-08-17T12:31:06.8756642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi80cf55 2022-08-17T12:31:06.8758099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi80cf55/_remote_module_non_scriptable.py 2022-08-17T12:31:06.8760574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoas4ejri 2022-08-17T12:31:06.8763126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoas4ejri/_remote_module_non_scriptable.py 2022-08-17T12:31:06.9380751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7a3totw 2022-08-17T12:31:06.9381582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7a3totw/_remote_module_non_scriptable.py 2022-08-17T12:31:07.1356185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:07.1373960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:07.1385962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:07.2017694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:07.5485569Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T12:31:07.5486199Z 2022-08-17T12:31:07.5486875Z ---------------------------------------------------------------------- 2022-08-17T12:31:07.5487320Z Ran 1 test in 2.075s 2022-08-17T12:31:07.5487516Z 2022-08-17T12:31:07.5487603Z OK (skipped=1) 2022-08-17T12:31:07.5487719Z 2022-08-17T12:31:07.5487812Z Generating XML reports... 2022-08-17T12:31:07.5526549Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123105.xml 2022-08-17T12:31:08.8505445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:08.8505841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:08.8507902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:08.8508278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:09.0011908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmale8t_ 2022-08-17T12:31:09.0015165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmale8t_/_remote_module_non_scriptable.py 2022-08-17T12:31:09.2598466Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:09.2609729Z 2022-08-17T12:31:09.2609943Z Running tests... 2022-08-17T12:31:09.2610325Z ---------------------------------------------------------------------- 2022-08-17T12:31:09.5318683Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27328 2022-08-17T12:31:09.5348309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27329 2022-08-17T12:31:09.5379564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27330 2022-08-17T12:31:09.5414719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27331 2022-08-17T12:31:10.6020952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:10.6021586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:10.6024239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:10.6024880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:10.6025667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:10.6026255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:10.6029194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:10.6029853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:10.6095139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:10.6095836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:10.6099794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:10.6100452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:10.6556850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:10.6557450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:10.6559925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:10.6560790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:10.7620373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_aw9iui 2022-08-17T12:31:10.7621220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1xdt7cn 2022-08-17T12:31:10.7621987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_aw9iui/_remote_module_non_scriptable.py 2022-08-17T12:31:10.7622772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1xdt7cn/_remote_module_non_scriptable.py 2022-08-17T12:31:10.7689078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjs8qmti 2022-08-17T12:31:10.7689997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjs8qmti/_remote_module_non_scriptable.py 2022-08-17T12:31:10.8136952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y3uc6ce 2022-08-17T12:31:10.8137798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y3uc6ce/_remote_module_non_scriptable.py 2022-08-17T12:31:11.0231094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:11.0239999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:11.0321850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:11.0950806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:11.4465124Z skip: Need at least 4 CUDA devices (2.185s) 2022-08-17T12:31:11.4465395Z 2022-08-17T12:31:11.4465930Z ---------------------------------------------------------------------- 2022-08-17T12:31:11.4466270Z Ran 1 test in 2.185s 2022-08-17T12:31:11.4466391Z 2022-08-17T12:31:11.4466472Z OK (skipped=1) 2022-08-17T12:31:11.4466585Z 2022-08-17T12:31:11.4466663Z Generating XML reports... 2022-08-17T12:31:11.4505720Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123109.xml 2022-08-17T12:31:12.7715689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:12.7716113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:12.7718886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:12.7719282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:12.9262571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h6c6ufd 2022-08-17T12:31:12.9265952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h6c6ufd/_remote_module_non_scriptable.py 2022-08-17T12:31:13.1928062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:13.1938801Z 2022-08-17T12:31:13.1938896Z Running tests... 2022-08-17T12:31:13.1939643Z ---------------------------------------------------------------------- 2022-08-17T12:31:13.4628998Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27383 2022-08-17T12:31:13.4658015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27384 2022-08-17T12:31:13.4687114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27385 2022-08-17T12:31:13.4719165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27386 2022-08-17T12:31:14.5399170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:14.5399807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:14.5402809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:14.5403773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:14.5442910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:14.5443561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:14.5458232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:14.5458895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:14.5891654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:14.5892243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:14.5895519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:14.5896215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:14.6403434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:14.6404423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:14.6407465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:14.6408067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:14.7048926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ht2xpuv 2022-08-17T12:31:14.7049820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ht2xpuv/_remote_module_non_scriptable.py 2022-08-17T12:31:14.7188107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fsu1k7g 2022-08-17T12:31:14.7189402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fsu1k7g/_remote_module_non_scriptable.py 2022-08-17T12:31:14.7527904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_fuwqc8 2022-08-17T12:31:14.7528788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_fuwqc8/_remote_module_non_scriptable.py 2022-08-17T12:31:14.7996459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_u14jxj 2022-08-17T12:31:14.7997461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_u14jxj/_remote_module_non_scriptable.py 2022-08-17T12:31:14.9698554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:14.9839628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:15.0245260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:15.0630837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:15.3772248Z skip: Need at least 4 CUDA devices (2.183s) 2022-08-17T12:31:15.3772521Z 2022-08-17T12:31:15.3772989Z ---------------------------------------------------------------------- 2022-08-17T12:31:15.3773403Z Ran 1 test in 2.183s 2022-08-17T12:31:15.3773610Z 2022-08-17T12:31:15.3773757Z OK (skipped=1) 2022-08-17T12:31:15.3773918Z 2022-08-17T12:31:15.3774049Z Generating XML reports... 2022-08-17T12:31:15.3810476Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123113.xml 2022-08-17T12:31:16.6783514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:16.6784179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:16.6785463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:16.6786342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:16.8308398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwg1n55c 2022-08-17T12:31:16.8310765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwg1n55c/_remote_module_non_scriptable.py 2022-08-17T12:31:17.0860569Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:17.0871878Z 2022-08-17T12:31:17.0872302Z Running tests... 2022-08-17T12:31:17.0872730Z ---------------------------------------------------------------------- 2022-08-17T12:31:17.3468289Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27438 2022-08-17T12:31:17.3497419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27439 2022-08-17T12:31:17.3526735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27440 2022-08-17T12:31:17.3557989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27441 2022-08-17T12:31:18.4220647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:18.4221613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:18.4223900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:18.4224575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:18.4308617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:18.4309294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:18.4312793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:18.4313491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:18.4695192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:18.4695773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:18.4698674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:18.4699241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:18.4775574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:18.4776143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:18.4780291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:18.4780994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:18.5807799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ci8xzqe 2022-08-17T12:31:18.5808693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ci8xzqe/_remote_module_non_scriptable.py 2022-08-17T12:31:18.5905092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u7qsh21 2022-08-17T12:31:18.5907280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u7qsh21/_remote_module_non_scriptable.py 2022-08-17T12:31:18.6271960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1y824a8 2022-08-17T12:31:18.6274175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1y824a8/_remote_module_non_scriptable.py 2022-08-17T12:31:18.6680210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4u9r1b9 2022-08-17T12:31:18.6681523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4u9r1b9/_remote_module_non_scriptable.py 2022-08-17T12:31:18.8401273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:18.8793329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:18.9029022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:18.9507793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:19.2610316Z skip: Need at least 2 CUDA devices (2.174s) 2022-08-17T12:31:19.2610495Z 2022-08-17T12:31:19.2610982Z ---------------------------------------------------------------------- 2022-08-17T12:31:19.2611475Z Ran 1 test in 2.174s 2022-08-17T12:31:19.2611676Z 2022-08-17T12:31:19.2611756Z OK (skipped=1) 2022-08-17T12:31:19.2611871Z 2022-08-17T12:31:19.2611947Z Generating XML reports... 2022-08-17T12:31:19.2649021Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123117.xml 2022-08-17T12:31:20.5779247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:20.5779650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:20.5781973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:20.5782373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:20.7286774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhu4j1j2 2022-08-17T12:31:20.7290225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhu4j1j2/_remote_module_non_scriptable.py 2022-08-17T12:31:20.9895018Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:20.9905441Z 2022-08-17T12:31:20.9905613Z Running tests... 2022-08-17T12:31:20.9906231Z ---------------------------------------------------------------------- 2022-08-17T12:31:21.2543914Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27493 2022-08-17T12:31:21.2572063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27494 2022-08-17T12:31:21.2601446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27495 2022-08-17T12:31:21.2632880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27496 2022-08-17T12:31:22.2723368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:22.2724052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:22.2726475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:22.2727106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:22.2858192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:22.2858858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:22.2863178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:22.2863868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:22.3123319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:22.3123987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:22.3127103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:22.3127699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:22.3145057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:22.3145673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:22.3149526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:22.3150219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:22.4301660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij5hu7un 2022-08-17T12:31:22.4302911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij5hu7un/_remote_module_non_scriptable.py 2022-08-17T12:31:22.4435770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vif2k35 2022-08-17T12:31:22.4436697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vif2k35/_remote_module_non_scriptable.py 2022-08-17T12:31:22.4714810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw6lr__0 2022-08-17T12:31:22.4715657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw6lr__0/_remote_module_non_scriptable.py 2022-08-17T12:31:22.4731818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus4hyxte 2022-08-17T12:31:22.4733098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus4hyxte/_remote_module_non_scriptable.py 2022-08-17T12:31:22.6892225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:22.7006413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:22.7305026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:22.7467128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:23.0682040Z skip: Need at least 1 CUDA device (2.077s) 2022-08-17T12:31:23.0682247Z 2022-08-17T12:31:23.0682738Z ---------------------------------------------------------------------- 2022-08-17T12:31:23.0683187Z Ran 1 test in 2.078s 2022-08-17T12:31:23.0683379Z 2022-08-17T12:31:23.0683501Z OK (skipped=1) 2022-08-17T12:31:23.0683617Z 2022-08-17T12:31:23.0683725Z Generating XML reports... 2022-08-17T12:31:23.0719644Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123120.xml 2022-08-17T12:31:24.3547145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:24.3547611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:24.3550022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:24.3550445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:24.5052149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd33onek 2022-08-17T12:31:24.5055630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd33onek/_remote_module_non_scriptable.py 2022-08-17T12:31:24.7631338Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:24.7641955Z 2022-08-17T12:31:24.7642056Z Running tests... 2022-08-17T12:31:24.7642532Z ---------------------------------------------------------------------- 2022-08-17T12:31:25.0287859Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27548 2022-08-17T12:31:25.0316354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27549 2022-08-17T12:31:25.0345798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27550 2022-08-17T12:31:25.0375427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27551 2022-08-17T12:31:26.1108673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:26.1109266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:26.1112419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:26.1113135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:26.1331010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:26.1331653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:26.1352799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:26.1353480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:26.1647700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:26.1648358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:26.1651896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:26.1652543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:26.1828141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:26.1828819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:26.1832395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:26.1833063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:26.2777923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputcs23hk 2022-08-17T12:31:26.2779594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputcs23hk/_remote_module_non_scriptable.py 2022-08-17T12:31:26.3065722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxkz0kxk 2022-08-17T12:31:26.3066778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxkz0kxk/_remote_module_non_scriptable.py 2022-08-17T12:31:26.3256770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhguroky 2022-08-17T12:31:26.3258047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhguroky/_remote_module_non_scriptable.py 2022-08-17T12:31:26.3438900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph070yv0m 2022-08-17T12:31:26.3439856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph070yv0m/_remote_module_non_scriptable.py 2022-08-17T12:31:26.5391859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:26.5683021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:26.5831294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:26.6118925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:26.9426619Z skip: Need at least 2 CUDA devices (2.178s) 2022-08-17T12:31:26.9426810Z 2022-08-17T12:31:26.9427168Z ---------------------------------------------------------------------- 2022-08-17T12:31:26.9427542Z Ran 1 test in 2.178s 2022-08-17T12:31:26.9427663Z 2022-08-17T12:31:26.9427727Z OK (skipped=1) 2022-08-17T12:31:26.9427839Z 2022-08-17T12:31:26.9427964Z Generating XML reports... 2022-08-17T12:31:26.9462885Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123124.xml 2022-08-17T12:31:28.2287921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:28.2288321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:28.2291361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:28.2291775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:28.3795207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kq6fw63 2022-08-17T12:31:28.3798239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kq6fw63/_remote_module_non_scriptable.py 2022-08-17T12:31:28.6359165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:28.6370391Z 2022-08-17T12:31:28.6370633Z Running tests... 2022-08-17T12:31:28.6371077Z ---------------------------------------------------------------------- 2022-08-17T12:31:28.9002687Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27603 2022-08-17T12:31:28.9031470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27604 2022-08-17T12:31:28.9060282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27605 2022-08-17T12:31:28.9091817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27606 2022-08-17T12:31:29.9550498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:29.9551187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:29.9553628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:29.9554274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:29.9559859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:29.9560525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:29.9565275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:29.9566143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:29.9573196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:29.9573872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:29.9577841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:29.9578451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:29.9579204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:29.9579794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:29.9582703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:29.9583369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:30.1167635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdiad4bi_ 2022-08-17T12:31:30.1168273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4m4osaat 2022-08-17T12:31:30.1168738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2432295q 2022-08-17T12:31:30.1169260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdiad4bi_/_remote_module_non_scriptable.py 2022-08-17T12:31:30.1169792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4m4osaat/_remote_module_non_scriptable.py 2022-08-17T12:31:30.1170605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2432295q/_remote_module_non_scriptable.py 2022-08-17T12:31:30.1178915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr745qfx1 2022-08-17T12:31:30.1181805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr745qfx1/_remote_module_non_scriptable.py 2022-08-17T12:31:30.3797854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:30.3858312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:30.3885593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:30.4038198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:30.7142432Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T12:31:30.7142789Z 2022-08-17T12:31:30.7143263Z ---------------------------------------------------------------------- 2022-08-17T12:31:30.7143506Z Ran 1 test in 2.077s 2022-08-17T12:31:30.7143626Z 2022-08-17T12:31:30.7143702Z OK (skipped=1) 2022-08-17T12:31:30.7143813Z 2022-08-17T12:31:30.7143903Z Generating XML reports... 2022-08-17T12:31:30.7180837Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123128.xml 2022-08-17T12:31:32.0195336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:32.0195736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:32.0197920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:32.0198504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:32.1694451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqna2w486 2022-08-17T12:31:32.1696603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqna2w486/_remote_module_non_scriptable.py 2022-08-17T12:31:32.4276725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:32.4287557Z 2022-08-17T12:31:32.4287700Z Running tests... 2022-08-17T12:31:32.4288140Z ---------------------------------------------------------------------- 2022-08-17T12:31:32.6907624Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27658 2022-08-17T12:31:32.6937764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27659 2022-08-17T12:31:32.6967296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27660 2022-08-17T12:31:32.6999562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27661 2022-08-17T12:31:33.7314808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:33.7315467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:33.7318366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:33.7318975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:33.7879124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:33.7879751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:33.7882261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:33.7882787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:33.7917408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:33.7918009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:33.7922315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:33.7922888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:33.8112107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:33.8112747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:33.8116683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:33.8117309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:33.8907307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z0uy3kd 2022-08-17T12:31:33.8908600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z0uy3kd/_remote_module_non_scriptable.py 2022-08-17T12:31:33.9464909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe381ph_o 2022-08-17T12:31:33.9466054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe381ph_o/_remote_module_non_scriptable.py 2022-08-17T12:31:33.9497680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rvv1_f8 2022-08-17T12:31:33.9498820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rvv1_f8/_remote_module_non_scriptable.py 2022-08-17T12:31:33.9709503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsndtszxx 2022-08-17T12:31:33.9710293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsndtszxx/_remote_module_non_scriptable.py 2022-08-17T12:31:34.1538415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:34.2066707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:34.2075425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:34.2337708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:34.6049675Z skip: Need at least 1 CUDA device (2.176s) 2022-08-17T12:31:34.6049901Z 2022-08-17T12:31:34.6050224Z ---------------------------------------------------------------------- 2022-08-17T12:31:34.6050547Z Ran 1 test in 2.176s 2022-08-17T12:31:34.6050667Z 2022-08-17T12:31:34.6050748Z OK (skipped=1) 2022-08-17T12:31:34.6050863Z 2022-08-17T12:31:34.6050952Z Generating XML reports... 2022-08-17T12:31:34.6086607Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123132.xml 2022-08-17T12:31:35.8831221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:35.8831621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:35.8833949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:35.8834369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:36.0326159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrr0szn2 2022-08-17T12:31:36.0329134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrr0szn2/_remote_module_non_scriptable.py 2022-08-17T12:31:36.2915078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:36.2926769Z 2022-08-17T12:31:36.2926873Z Running tests... 2022-08-17T12:31:36.2927480Z ---------------------------------------------------------------------- 2022-08-17T12:31:36.5576513Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27713 2022-08-17T12:31:36.5605209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27714 2022-08-17T12:31:36.5635771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27715 2022-08-17T12:31:36.5664703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27716 2022-08-17T12:31:37.5651333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:37.5651737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:37.5680051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:37.5680703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:37.5770455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:37.5771068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:37.5776023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:37.5776690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:37.6028082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:37.6028766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:37.6030910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:37.6031554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:37.6111679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:37.6112347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:37.6115629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:37.6116282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:37.7254252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxibj8b6k 2022-08-17T12:31:37.7255036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxibj8b6k/_remote_module_non_scriptable.py 2022-08-17T12:31:37.7337934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8reqym9z 2022-08-17T12:31:37.7338786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8reqym9z/_remote_module_non_scriptable.py 2022-08-17T12:31:37.7613722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9x1t4nk8 2022-08-17T12:31:37.7614961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9x1t4nk8/_remote_module_non_scriptable.py 2022-08-17T12:31:37.7686575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaen4jaau 2022-08-17T12:31:37.7687834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaen4jaau/_remote_module_non_scriptable.py 2022-08-17T12:31:37.9850254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:37.9919730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:38.0285995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:38.0379029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:38.3714707Z skip: Need at least 2 CUDA devices (2.079s) 2022-08-17T12:31:38.3715007Z 2022-08-17T12:31:38.3715612Z ---------------------------------------------------------------------- 2022-08-17T12:31:38.3715862Z Ran 1 test in 2.079s 2022-08-17T12:31:38.3715994Z 2022-08-17T12:31:38.3716072Z OK (skipped=1) 2022-08-17T12:31:38.3716185Z 2022-08-17T12:31:38.3716276Z Generating XML reports... 2022-08-17T12:31:38.3752810Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123136.xml 2022-08-17T12:31:39.6585449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:39.6585851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:39.6588373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:39.6588737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:39.8107921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyux1bt0m 2022-08-17T12:31:39.8111328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyux1bt0m/_remote_module_non_scriptable.py 2022-08-17T12:31:40.0696968Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:40.0707825Z 2022-08-17T12:31:40.0708306Z Running tests... 2022-08-17T12:31:40.0708672Z ---------------------------------------------------------------------- 2022-08-17T12:31:40.3344353Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27768 2022-08-17T12:31:40.3373302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27769 2022-08-17T12:31:40.3403598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27770 2022-08-17T12:31:40.3436908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27771 2022-08-17T12:31:41.3542070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:41.3542694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:41.3545397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:41.3546047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:41.3740489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:41.3741142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:41.3744308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:41.3745193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:41.3781353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:41.3781985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:41.3785758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:41.3786427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:41.4175560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:41.4176257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:41.4177856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:41.4178564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:41.5140238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_5r8pmz 2022-08-17T12:31:41.5141422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_5r8pmz/_remote_module_non_scriptable.py 2022-08-17T12:31:41.5330125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlu3y1l1 2022-08-17T12:31:41.5331007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlu3y1l1/_remote_module_non_scriptable.py 2022-08-17T12:31:41.5396088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gn0kupo 2022-08-17T12:31:41.5397664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gn0kupo/_remote_module_non_scriptable.py 2022-08-17T12:31:41.5739893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplukvfk5q 2022-08-17T12:31:41.5740760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplukvfk5q/_remote_module_non_scriptable.py 2022-08-17T12:31:41.7967805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:41.7973995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:41.8029682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:41.8387577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:42.1487203Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T12:31:42.1487434Z 2022-08-17T12:31:42.1487743Z ---------------------------------------------------------------------- 2022-08-17T12:31:42.1488003Z Ran 1 test in 2.078s 2022-08-17T12:31:42.1488125Z 2022-08-17T12:31:42.1488200Z OK (skipped=1) 2022-08-17T12:31:42.1488312Z 2022-08-17T12:31:42.1488404Z Generating XML reports... 2022-08-17T12:31:42.1524947Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123140.xml 2022-08-17T12:31:43.4480346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:43.4480754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:43.4483163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:43.4483607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:43.5981296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6t5aggx 2022-08-17T12:31:43.5984296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6t5aggx/_remote_module_non_scriptable.py 2022-08-17T12:31:43.8550069Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:43.8560939Z 2022-08-17T12:31:43.8561340Z Running tests... 2022-08-17T12:31:43.8561765Z ---------------------------------------------------------------------- 2022-08-17T12:31:44.1162007Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27823 2022-08-17T12:31:44.1190030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27824 2022-08-17T12:31:44.1218626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27825 2022-08-17T12:31:44.1249101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27826 2022-08-17T12:31:45.1715872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:45.1716529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:45.1719007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:45.1719631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:45.1752433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:45.1753363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:45.1756677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:45.1757310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:45.1800686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:45.1801307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:45.1806231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:45.1806876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:45.1991403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:45.1992112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:45.1995166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:45.1996036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:45.3305706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa7___xt 2022-08-17T12:31:45.3306671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa7___xt/_remote_module_non_scriptable.py 2022-08-17T12:31:45.3337161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9muyugqa 2022-08-17T12:31:45.3339125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9muyugqa/_remote_module_non_scriptable.py 2022-08-17T12:31:45.3396169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2trscv5 2022-08-17T12:31:45.3398050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2trscv5/_remote_module_non_scriptable.py 2022-08-17T12:31:45.3564594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm8vsyv4 2022-08-17T12:31:45.3565367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm8vsyv4/_remote_module_non_scriptable.py 2022-08-17T12:31:45.5883343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:45.5903971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:45.5992260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:45.6146790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:45.9299886Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T12:31:45.9300124Z 2022-08-17T12:31:45.9300654Z ---------------------------------------------------------------------- 2022-08-17T12:31:45.9301098Z Ran 1 test in 2.074s 2022-08-17T12:31:45.9301207Z 2022-08-17T12:31:45.9301286Z OK (skipped=1) 2022-08-17T12:31:45.9301403Z 2022-08-17T12:31:45.9301495Z Generating XML reports... 2022-08-17T12:31:45.9337723Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123143.xml 2022-08-17T12:31:47.2456172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:47.2456574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:47.2458996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:47.2459468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:47.3963370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a3nkoci 2022-08-17T12:31:47.3966805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a3nkoci/_remote_module_non_scriptable.py 2022-08-17T12:31:47.6531568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:47.6541982Z 2022-08-17T12:31:47.6542459Z Running tests... 2022-08-17T12:31:47.6542903Z ---------------------------------------------------------------------- 2022-08-17T12:31:47.9164053Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27878 2022-08-17T12:31:47.9192470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27879 2022-08-17T12:31:47.9222032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27880 2022-08-17T12:31:47.9251895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27881 2022-08-17T12:31:48.8891282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:48.8891941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:48.8894066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:48.8895583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:48.9418441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:48.9418990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:48.9422266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:48.9422942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:48.9812638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:48.9813308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:48.9816204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:48.9816859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:48.9883809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:48.9884493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:48.9888915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:48.9889485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:49.0662710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2k9rj25 2022-08-17T12:31:49.0666140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2k9rj25/_remote_module_non_scriptable.py 2022-08-17T12:31:49.1018074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yy2pre_ 2022-08-17T12:31:49.1018941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yy2pre_/_remote_module_non_scriptable.py 2022-08-17T12:31:49.1413308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg24_lan 2022-08-17T12:31:49.1414148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg24_lan/_remote_module_non_scriptable.py 2022-08-17T12:31:49.1528039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u0rucv5 2022-08-17T12:31:49.1528790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u0rucv5/_remote_module_non_scriptable.py 2022-08-17T12:31:49.3264303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:49.3638428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:49.4019213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:49.4376895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:49.7302985Z skip: Need at least 1 CUDA device (2.076s) 2022-08-17T12:31:49.7303174Z 2022-08-17T12:31:49.7303588Z ---------------------------------------------------------------------- 2022-08-17T12:31:49.7303935Z Ran 1 test in 2.076s 2022-08-17T12:31:49.7304055Z 2022-08-17T12:31:49.7304132Z OK (skipped=1) 2022-08-17T12:31:49.7304247Z 2022-08-17T12:31:49.7304338Z Generating XML reports... 2022-08-17T12:31:49.7340030Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123147.xml 2022-08-17T12:31:51.0279951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:51.0280367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:51.0282736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:51.0283373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:51.1828870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b09hh23 2022-08-17T12:31:51.1831920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b09hh23/_remote_module_non_scriptable.py 2022-08-17T12:31:51.4410205Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:51.4420203Z 2022-08-17T12:31:51.4420354Z Running tests... 2022-08-17T12:31:51.4420988Z ---------------------------------------------------------------------- 2022-08-17T12:31:51.7052988Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27933 2022-08-17T12:31:51.7081359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27934 2022-08-17T12:31:51.7109201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27935 2022-08-17T12:31:51.7141671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27936 2022-08-17T12:31:52.8037183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:52.8037908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:52.8041565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:52.8042249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:52.8275414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:52.8276132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:52.8278945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:52.8279626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:52.8304977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:52.8305593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:52.8310028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:52.8310626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:52.8313392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:52.8314280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:52.8318710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:52.8319312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:52.9630611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_tzi8fs 2022-08-17T12:31:52.9631476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_tzi8fs/_remote_module_non_scriptable.py 2022-08-17T12:31:52.9901093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf5quw7q 2022-08-17T12:31:52.9902376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf5quw7q/_remote_module_non_scriptable.py 2022-08-17T12:31:52.9905089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeopzcx1p 2022-08-17T12:31:52.9907755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeopzcx1p/_remote_module_non_scriptable.py 2022-08-17T12:31:52.9944893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5elbity 2022-08-17T12:31:52.9946892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5elbity/_remote_module_non_scriptable.py 2022-08-17T12:31:53.2275605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:53.2533346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:53.2615500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:53.2732417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:53.6193772Z skip: Need at least 2 CUDA devices (2.177s) 2022-08-17T12:31:53.6194088Z 2022-08-17T12:31:53.6194525Z ---------------------------------------------------------------------- 2022-08-17T12:31:53.6194806Z Ran 1 test in 2.177s 2022-08-17T12:31:53.6194913Z 2022-08-17T12:31:53.6194990Z OK (skipped=1) 2022-08-17T12:31:53.6195105Z 2022-08-17T12:31:53.6195197Z Generating XML reports... 2022-08-17T12:31:53.6231005Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123151.xml 2022-08-17T12:31:54.9212872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:54.9213267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:54.9215698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:54.9216060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:55.0772313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz36_vn0f 2022-08-17T12:31:55.0775500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz36_vn0f/_remote_module_non_scriptable.py 2022-08-17T12:31:55.3383396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:55.3394778Z 2022-08-17T12:31:55.3394938Z Running tests... 2022-08-17T12:31:55.3395366Z ---------------------------------------------------------------------- 2022-08-17T12:31:55.5883758Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.248s) 2022-08-17T12:31:55.5884411Z 2022-08-17T12:31:55.5884633Z ---------------------------------------------------------------------- 2022-08-17T12:31:55.5884891Z Ran 1 test in 0.249s 2022-08-17T12:31:55.5885213Z 2022-08-17T12:31:55.5885289Z OK (skipped=1) 2022-08-17T12:31:55.5885405Z 2022-08-17T12:31:55.5885494Z Generating XML reports... 2022-08-17T12:31:55.5907291Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123155.xml 2022-08-17T12:31:56.7991887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:56.7992268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:56.7995345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:56.7995742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:56.9493166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd37aw_w6 2022-08-17T12:31:56.9496786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd37aw_w6/_remote_module_non_scriptable.py 2022-08-17T12:31:57.2043770Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:31:57.2054288Z 2022-08-17T12:31:57.2054438Z Running tests... 2022-08-17T12:31:57.2055092Z ---------------------------------------------------------------------- 2022-08-17T12:31:57.4674269Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27998 2022-08-17T12:31:57.4701933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27999 2022-08-17T12:31:57.4730769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28000 2022-08-17T12:31:57.4762205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28001 2022-08-17T12:31:58.5399846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:58.5400586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:58.5403315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:58.5403968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:58.5404703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:58.5405332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:58.5407013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:58.5407593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:58.5557915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:58.5558569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:58.5561811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:58.5562476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:58.6019626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:31:58.6020209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:31:58.6022855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:31:58.6023377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:31:58.6990156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbnllx14 2022-08-17T12:31:58.6991009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbnllx14/_remote_module_non_scriptable.py 2022-08-17T12:31:58.7001544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb90_tff 2022-08-17T12:31:58.7003039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb90_tff/_remote_module_non_scriptable.py 2022-08-17T12:31:58.7121252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnlu4dr3 2022-08-17T12:31:58.7122513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnlu4dr3/_remote_module_non_scriptable.py 2022-08-17T12:31:58.7568648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpm9hztm 2022-08-17T12:31:58.7569933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpm9hztm/_remote_module_non_scriptable.py 2022-08-17T12:31:58.9574907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:31:58.9585257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:31:58.9737236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:31:59.0165030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:31:59.2811379Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T12:31:59.2811749Z 2022-08-17T12:31:59.2812586Z ---------------------------------------------------------------------- 2022-08-17T12:31:59.2812859Z Ran 1 test in 2.076s 2022-08-17T12:31:59.2812985Z 2022-08-17T12:31:59.2813063Z OK (skipped=1) 2022-08-17T12:31:59.2813163Z 2022-08-17T12:31:59.2813310Z Generating XML reports... 2022-08-17T12:31:59.2848658Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123157.xml 2022-08-17T12:32:00.5654282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:00.5654677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:00.5657086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:00.5657605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:00.7142846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iecbmfr 2022-08-17T12:32:00.7145707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iecbmfr/_remote_module_non_scriptable.py 2022-08-17T12:32:00.9711166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:00.9720951Z 2022-08-17T12:32:00.9721077Z Running tests... 2022-08-17T12:32:00.9721732Z ---------------------------------------------------------------------- 2022-08-17T12:32:01.2377485Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28053 2022-08-17T12:32:01.2406795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28054 2022-08-17T12:32:01.2435873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28055 2022-08-17T12:32:01.2467171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28056 2022-08-17T12:32:02.2534376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:02.2535026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:02.2577952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:02.2578667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:02.2922887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:02.2923601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:02.2924960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:02.2925620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:02.2926709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:02.2927395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:02.2930411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:02.2931038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:02.3246155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:02.3246852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:02.3249381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:02.3250046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:02.4141882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nvc1vit 2022-08-17T12:32:02.4142868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nvc1vit/_remote_module_non_scriptable.py 2022-08-17T12:32:02.4515045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpca7kw2oq 2022-08-17T12:32:02.4515912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpca7kw2oq/_remote_module_non_scriptable.py 2022-08-17T12:32:02.4518843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvy9j5r0 2022-08-17T12:32:02.4521416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvy9j5r0/_remote_module_non_scriptable.py 2022-08-17T12:32:02.4873418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89eou4ui 2022-08-17T12:32:02.4874637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89eou4ui/_remote_module_non_scriptable.py 2022-08-17T12:32:02.6830561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:02.7149344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:02.7168124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:02.7675359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:03.0517224Z skip: Need at least 1 CUDA device (2.079s) 2022-08-17T12:32:03.0517497Z 2022-08-17T12:32:03.0517976Z ---------------------------------------------------------------------- 2022-08-17T12:32:03.0518361Z Ran 1 test in 2.080s 2022-08-17T12:32:03.0518548Z 2022-08-17T12:32:03.0518664Z OK (skipped=1) 2022-08-17T12:32:03.0518842Z 2022-08-17T12:32:03.0518980Z Generating XML reports... 2022-08-17T12:32:03.0555764Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123200.xml 2022-08-17T12:32:04.3618215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:04.3618937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:04.3621672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:04.3622261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:04.5121848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sx0xdhb 2022-08-17T12:32:04.5124729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sx0xdhb/_remote_module_non_scriptable.py 2022-08-17T12:32:04.7720837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:04.7732605Z 2022-08-17T12:32:04.7733090Z Running tests... 2022-08-17T12:32:04.7733531Z ---------------------------------------------------------------------- 2022-08-17T12:32:05.0416791Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28108 2022-08-17T12:32:05.0445815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28109 2022-08-17T12:32:05.0475612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28110 2022-08-17T12:32:05.0509821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28111 2022-08-17T12:32:06.1027873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:06.1030368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:06.1031295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:06.1031921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:06.1257774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:06.1258477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:06.1261508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:06.1262134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:06.1598771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:06.1599446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:06.1603198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:06.1603874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:06.1631072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:06.1631653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:06.1636161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:06.1636775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:06.2615703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperi0mtmt 2022-08-17T12:32:06.2616577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperi0mtmt/_remote_module_non_scriptable.py 2022-08-17T12:32:06.2826453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoik35jn_ 2022-08-17T12:32:06.2827266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoik35jn_/_remote_module_non_scriptable.py 2022-08-17T12:32:06.3196067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp013vtqeo 2022-08-17T12:32:06.3197363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp013vtqeo/_remote_module_non_scriptable.py 2022-08-17T12:32:06.3235342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rdqm2si 2022-08-17T12:32:06.3237936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rdqm2si/_remote_module_non_scriptable.py 2022-08-17T12:32:06.5252952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:06.5428822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:06.5838764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:06.5853154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:06.9560928Z skip: Need at least 1 CUDA device (2.182s) 2022-08-17T12:32:06.9561121Z 2022-08-17T12:32:06.9561438Z ---------------------------------------------------------------------- 2022-08-17T12:32:06.9561816Z Ran 1 test in 2.183s 2022-08-17T12:32:06.9561939Z 2022-08-17T12:32:06.9562003Z OK (skipped=1) 2022-08-17T12:32:06.9562115Z 2022-08-17T12:32:06.9562205Z Generating XML reports... 2022-08-17T12:32:06.9597975Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123204.xml 2022-08-17T12:32:08.2818434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:08.2818814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:08.2821326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:08.2821719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:08.4363203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08_rd639 2022-08-17T12:32:08.4367353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08_rd639/_remote_module_non_scriptable.py 2022-08-17T12:32:08.6977542Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:08.6988605Z 2022-08-17T12:32:08.6988708Z Running tests... 2022-08-17T12:32:08.6989163Z ---------------------------------------------------------------------- 2022-08-17T12:32:08.9681807Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28163 2022-08-17T12:32:08.9712039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28164 2022-08-17T12:32:08.9742103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28165 2022-08-17T12:32:08.9773275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28166 2022-08-17T12:32:10.0136110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:10.0136733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:10.0139749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:10.0140323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:10.0173131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:10.0173771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:10.0177341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:10.0178164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:10.0453969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:10.0454642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:10.0457861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:10.0458544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:10.0511625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:10.0512253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:10.0515939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:10.0516884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:10.1748164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgt3ulj1c 2022-08-17T12:32:10.1749221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgt3ulj1c/_remote_module_non_scriptable.py 2022-08-17T12:32:10.1767483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql_2d4qs 2022-08-17T12:32:10.1768816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql_2d4qs/_remote_module_non_scriptable.py 2022-08-17T12:32:10.2037029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lf51_01 2022-08-17T12:32:10.2037815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lf51_01/_remote_module_non_scriptable.py 2022-08-17T12:32:10.2104402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39x_405q 2022-08-17T12:32:10.2105723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39x_405q/_remote_module_non_scriptable.py 2022-08-17T12:32:10.4374534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:10.4375807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:10.4606290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:10.4792571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:10.7824836Z skip: Need at least 2 CUDA devices (2.083s) 2022-08-17T12:32:10.7825134Z 2022-08-17T12:32:10.7825469Z ---------------------------------------------------------------------- 2022-08-17T12:32:10.7825719Z Ran 1 test in 2.083s 2022-08-17T12:32:10.7825843Z 2022-08-17T12:32:10.7825934Z OK (skipped=1) 2022-08-17T12:32:10.7826046Z 2022-08-17T12:32:10.7826137Z Generating XML reports... 2022-08-17T12:32:10.7861098Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123208.xml 2022-08-17T12:32:12.0965750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:12.0966330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:12.0969075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:12.0969453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:12.2474413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqvmy7d5 2022-08-17T12:32:12.2478146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqvmy7d5/_remote_module_non_scriptable.py 2022-08-17T12:32:12.5040837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:12.5052634Z 2022-08-17T12:32:12.5052925Z Running tests... 2022-08-17T12:32:12.5053615Z ---------------------------------------------------------------------- 2022-08-17T12:32:12.7646971Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28218 2022-08-17T12:32:12.7675132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28219 2022-08-17T12:32:12.7703365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28220 2022-08-17T12:32:12.7735315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28221 2022-08-17T12:32:13.8118836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:13.8119502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:13.8122473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:13.8123525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:13.8127803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:13.8128423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:13.8132554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:13.8133143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:13.8388685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:13.8389340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:13.8392281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:13.8392946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:13.8568648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:13.8569619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:13.8573141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:13.8573801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:13.9718016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0lt_rb6 2022-08-17T12:32:13.9718837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0lt_rb6/_remote_module_non_scriptable.py 2022-08-17T12:32:13.9814489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel5z_qf4 2022-08-17T12:32:13.9815531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel5z_qf4/_remote_module_non_scriptable.py 2022-08-17T12:32:13.9973071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthzwpbo9 2022-08-17T12:32:13.9975277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthzwpbo9/_remote_module_non_scriptable.py 2022-08-17T12:32:14.0153816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ys0ukcj 2022-08-17T12:32:14.0154626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ys0ukcj/_remote_module_non_scriptable.py 2022-08-17T12:32:14.2341929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:14.2530521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:14.2605678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:14.2740209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:14.5786471Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T12:32:14.5786740Z 2022-08-17T12:32:14.5787279Z ---------------------------------------------------------------------- 2022-08-17T12:32:14.5787712Z Ran 1 test in 2.073s 2022-08-17T12:32:14.5787843Z 2022-08-17T12:32:14.5787920Z OK (skipped=1) 2022-08-17T12:32:14.5788021Z 2022-08-17T12:32:14.5788109Z Generating XML reports... 2022-08-17T12:32:14.5823109Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123212.xml 2022-08-17T12:32:15.8808316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:15.8808779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:15.8810205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:15.8810836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:16.0332609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3s7dsh_7 2022-08-17T12:32:16.0335787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3s7dsh_7/_remote_module_non_scriptable.py 2022-08-17T12:32:16.2944300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:16.2955331Z 2022-08-17T12:32:16.2955605Z Running tests... 2022-08-17T12:32:16.2955976Z ---------------------------------------------------------------------- 2022-08-17T12:32:16.5583659Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28273 2022-08-17T12:32:16.5611870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28274 2022-08-17T12:32:16.5640194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28275 2022-08-17T12:32:16.5671159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28276 2022-08-17T12:32:17.6575637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:17.6576600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:17.6578737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:17.6602254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:17.7198097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:17.7198731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:17.7201492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:17.7202110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:17.7221493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:17.7222189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:17.7226006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:17.7226641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:17.7966234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:17.7966884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:17.7970183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:17.7970877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:17.8150905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3svq9jm 2022-08-17T12:32:17.8151939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3svq9jm/_remote_module_non_scriptable.py 2022-08-17T12:32:17.8766027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8ptnnh8 2022-08-17T12:32:17.8766875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8ptnnh8/_remote_module_non_scriptable.py 2022-08-17T12:32:17.8804772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03wbak71 2022-08-17T12:32:17.8807031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03wbak71/_remote_module_non_scriptable.py 2022-08-17T12:32:17.9540542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplci8w8fq 2022-08-17T12:32:17.9541469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplci8w8fq/_remote_module_non_scriptable.py 2022-08-17T12:32:18.0774810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:18.1423826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:18.1629400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:18.2418126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:18.5724031Z skip: Need at least 2 CUDA devices (2.277s) 2022-08-17T12:32:18.5724487Z 2022-08-17T12:32:18.5724925Z ---------------------------------------------------------------------- 2022-08-17T12:32:18.5725176Z Ran 1 test in 2.277s 2022-08-17T12:32:18.5725296Z 2022-08-17T12:32:18.5725373Z OK (skipped=1) 2022-08-17T12:32:18.5725491Z 2022-08-17T12:32:18.5725583Z Generating XML reports... 2022-08-17T12:32:18.5761627Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123216.xml 2022-08-17T12:32:19.8768513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:19.8769106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:19.8771547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:19.8771914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:20.0301092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd69vg8uo 2022-08-17T12:32:20.0304729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd69vg8uo/_remote_module_non_scriptable.py 2022-08-17T12:32:20.2912231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:20.2923164Z 2022-08-17T12:32:20.2923612Z Running tests... 2022-08-17T12:32:20.2924042Z ---------------------------------------------------------------------- 2022-08-17T12:32:20.5567951Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28328 2022-08-17T12:32:20.5597995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28329 2022-08-17T12:32:20.5628675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28330 2022-08-17T12:32:20.5661171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28331 2022-08-17T12:32:21.5793299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:21.5793962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:21.5797592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:21.5798255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:21.6096522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:21.6097209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:21.6100275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:21.6100936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:21.6330365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:21.6331030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:21.6333880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:21.6334896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:21.6453850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:21.6454546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:21.6458006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:21.6458633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:21.7390855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmv8nsdy 2022-08-17T12:32:21.7391618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmv8nsdy/_remote_module_non_scriptable.py 2022-08-17T12:32:21.7674000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g2407r4 2022-08-17T12:32:21.7674846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g2407r4/_remote_module_non_scriptable.py 2022-08-17T12:32:21.7924151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95z6vz96 2022-08-17T12:32:21.7925532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95z6vz96/_remote_module_non_scriptable.py 2022-08-17T12:32:21.8058471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6d22atz 2022-08-17T12:32:21.8059327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6d22atz/_remote_module_non_scriptable.py 2022-08-17T12:32:22.0054502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:22.0308389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:22.0582260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:22.1015559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:22.3711372Z skip: Need at least 1 CUDA device (2.078s) 2022-08-17T12:32:22.3711695Z 2022-08-17T12:32:22.3712228Z ---------------------------------------------------------------------- 2022-08-17T12:32:22.3712495Z Ran 1 test in 2.079s 2022-08-17T12:32:22.3712617Z 2022-08-17T12:32:22.3712692Z OK (skipped=1) 2022-08-17T12:32:22.3712818Z 2022-08-17T12:32:22.3712895Z Generating XML reports... 2022-08-17T12:32:22.3748207Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123220.xml 2022-08-17T12:32:23.6861228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:23.6861823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:23.6864713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:23.6865329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:23.8369509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60imbtdl 2022-08-17T12:32:23.8372229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60imbtdl/_remote_module_non_scriptable.py 2022-08-17T12:32:24.0917707Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:24.0928863Z 2022-08-17T12:32:24.0929222Z Running tests... 2022-08-17T12:32:24.0929632Z ---------------------------------------------------------------------- 2022-08-17T12:32:24.3509572Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28383 2022-08-17T12:32:24.3537927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28384 2022-08-17T12:32:24.3567088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28385 2022-08-17T12:32:24.3598347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28386 2022-08-17T12:32:25.3567153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:25.3567965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:25.3570734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:25.3571381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:25.3584914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:25.3585638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:25.3589347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:25.3590020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:25.4022850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:25.4023435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:25.4028424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:25.4029032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:25.4157444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:25.4157984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:25.4161505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:25.4162056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:25.5216039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ozx6wpb 2022-08-17T12:32:25.5216777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ozx6wpb/_remote_module_non_scriptable.py 2022-08-17T12:32:25.5228433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78ivyuc4 2022-08-17T12:32:25.5230749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78ivyuc4/_remote_module_non_scriptable.py 2022-08-17T12:32:25.5809920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmi8043p 2022-08-17T12:32:25.5810892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmi8043p/_remote_module_non_scriptable.py 2022-08-17T12:32:25.5820748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g4pq4ik 2022-08-17T12:32:25.5822339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g4pq4ik/_remote_module_non_scriptable.py 2022-08-17T12:32:25.7884466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:25.7896456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:25.8490144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:25.8498051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:26.1649552Z skip: Need at least 2 CUDA devices (2.072s) 2022-08-17T12:32:26.1649810Z 2022-08-17T12:32:26.1650342Z ---------------------------------------------------------------------- 2022-08-17T12:32:26.1650803Z Ran 1 test in 2.072s 2022-08-17T12:32:26.1650941Z 2022-08-17T12:32:26.1651010Z OK (skipped=1) 2022-08-17T12:32:26.1651123Z 2022-08-17T12:32:26.1651214Z Generating XML reports... 2022-08-17T12:32:26.1686244Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123224.xml 2022-08-17T12:32:27.4764880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:27.4765313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:27.4767973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:27.4768334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:27.6387887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz05iz_2u 2022-08-17T12:32:27.6391034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz05iz_2u/_remote_module_non_scriptable.py 2022-08-17T12:32:27.8944231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:27.8955176Z 2022-08-17T12:32:27.8955333Z Running tests... 2022-08-17T12:32:27.8955791Z ---------------------------------------------------------------------- 2022-08-17T12:32:28.1582098Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28438 2022-08-17T12:32:28.1612082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28439 2022-08-17T12:32:28.1641316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28440 2022-08-17T12:32:28.1671063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28441 2022-08-17T12:32:29.1364496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:29.1365195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:29.1368290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:29.1368954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:29.1592479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:29.1593203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:29.1596890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:29.1597555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:29.1703801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:29.1704465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:29.1708157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:29.1708837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:29.2049316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:29.2050036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:29.2052173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:29.2052792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:29.2946462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g0vxrpr 2022-08-17T12:32:29.2947614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g0vxrpr/_remote_module_non_scriptable.py 2022-08-17T12:32:29.3188919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc2kw_vr 2022-08-17T12:32:29.3190997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc2kw_vr/_remote_module_non_scriptable.py 2022-08-17T12:32:29.3307746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rceona_ 2022-08-17T12:32:29.3308908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rceona_/_remote_module_non_scriptable.py 2022-08-17T12:32:29.3635684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmntr87h 2022-08-17T12:32:29.3636516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmntr87h/_remote_module_non_scriptable.py 2022-08-17T12:32:29.5560601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:29.5805667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:29.6119337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:29.6271539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:29.9721961Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T12:32:29.9722152Z 2022-08-17T12:32:29.9722473Z ---------------------------------------------------------------------- 2022-08-17T12:32:29.9722833Z Ran 1 test in 2.077s 2022-08-17T12:32:29.9722956Z 2022-08-17T12:32:29.9723034Z OK (skipped=1) 2022-08-17T12:32:29.9723149Z 2022-08-17T12:32:29.9723532Z Generating XML reports... 2022-08-17T12:32:29.9758799Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123227.xml 2022-08-17T12:32:31.2729955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:31.2730330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:31.2733644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:31.2734024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:31.4255215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_2_rlpj 2022-08-17T12:32:31.4258496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_2_rlpj/_remote_module_non_scriptable.py 2022-08-17T12:32:31.6874267Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:31.6889379Z 2022-08-17T12:32:31.6889509Z Running tests... 2022-08-17T12:32:31.6890227Z ---------------------------------------------------------------------- 2022-08-17T12:32:31.9500180Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28493 2022-08-17T12:32:31.9530516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28494 2022-08-17T12:32:31.9559237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28495 2022-08-17T12:32:31.9593355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28496 2022-08-17T12:32:32.9780475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:32.9781172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:32.9783898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:32.9784521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:32.9910583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:32.9911298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:32.9915083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:32.9915738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:32.9943315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:32.9943950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:32.9946765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:32.9947491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:33.0034960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:33.0035587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:33.0039625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:33.0040281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:33.1369504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu8dija4 2022-08-17T12:32:33.1370610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu8dija4/_remote_module_non_scriptable.py 2022-08-17T12:32:33.1500107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gxhcazv 2022-08-17T12:32:33.1500939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gxhcazv/_remote_module_non_scriptable.py 2022-08-17T12:32:33.1533958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq58akn0t 2022-08-17T12:32:33.1535502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq58akn0t/_remote_module_non_scriptable.py 2022-08-17T12:32:33.1601634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzhx2xiy 2022-08-17T12:32:33.1603645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzhx2xiy/_remote_module_non_scriptable.py 2022-08-17T12:32:33.3975758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:33.4093076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:33.4126762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:33.4163854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:33.7643530Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T12:32:33.7643830Z 2022-08-17T12:32:33.7644191Z ---------------------------------------------------------------------- 2022-08-17T12:32:33.7644516Z Ran 1 test in 2.075s 2022-08-17T12:32:33.7644648Z 2022-08-17T12:32:33.7644719Z OK (skipped=1) 2022-08-17T12:32:33.7644834Z 2022-08-17T12:32:33.7644922Z Generating XML reports... 2022-08-17T12:32:33.7696753Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123231.xml 2022-08-17T12:32:35.0515563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:35.0515965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:35.0518010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:35.0518385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:35.2029553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74z860u5 2022-08-17T12:32:35.2032566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74z860u5/_remote_module_non_scriptable.py 2022-08-17T12:32:35.4619419Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:35.4630450Z 2022-08-17T12:32:35.4630596Z Running tests... 2022-08-17T12:32:35.4631053Z ---------------------------------------------------------------------- 2022-08-17T12:32:35.7267443Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28548 2022-08-17T12:32:35.7296600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28549 2022-08-17T12:32:35.7325558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28550 2022-08-17T12:32:35.7356277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28551 2022-08-17T12:32:36.8210258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:36.8210939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:36.8214328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:36.8214997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:36.8215859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:36.8216474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:36.8219527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:36.8220226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:36.8476483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:36.8477145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:36.8478767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:36.8479389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:36.8522277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:36.8522893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:36.8527529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:36.8528168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:36.9851450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l0gi7yz 2022-08-17T12:32:36.9852227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7w1ru7g 2022-08-17T12:32:36.9852966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l0gi7yz/_remote_module_non_scriptable.py 2022-08-17T12:32:36.9853747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7w1ru7g/_remote_module_non_scriptable.py 2022-08-17T12:32:37.0130098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh_lfwa2 2022-08-17T12:32:37.0131132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh_lfwa2/_remote_module_non_scriptable.py 2022-08-17T12:32:37.0280565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegzacz6j 2022-08-17T12:32:37.0281426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegzacz6j/_remote_module_non_scriptable.py 2022-08-17T12:32:37.2490869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:37.2498801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:37.2838992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:37.2918799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:37.6407890Z skip: Need at least 1 CUDA device (2.177s) 2022-08-17T12:32:37.6408351Z 2022-08-17T12:32:37.6408748Z ---------------------------------------------------------------------- 2022-08-17T12:32:37.6409029Z Ran 1 test in 2.178s 2022-08-17T12:32:37.6409148Z 2022-08-17T12:32:37.6409211Z OK (skipped=1) 2022-08-17T12:32:37.6409405Z 2022-08-17T12:32:37.6409498Z Generating XML reports... 2022-08-17T12:32:37.6444711Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123235.xml 2022-08-17T12:32:38.9490591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:38.9490985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:38.9493617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:38.9493998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:39.1036502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxlww0hm 2022-08-17T12:32:39.1040111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxlww0hm/_remote_module_non_scriptable.py 2022-08-17T12:32:39.3652862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:39.3664259Z 2022-08-17T12:32:39.3664634Z Running tests... 2022-08-17T12:32:39.3665049Z ---------------------------------------------------------------------- 2022-08-17T12:32:39.6295669Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28603 2022-08-17T12:32:39.6322813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28604 2022-08-17T12:32:39.6351407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28605 2022-08-17T12:32:39.6381389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28606 2022-08-17T12:32:40.6832339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:40.6833039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:40.6835630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:40.6836254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:40.7608844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:40.7609463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:40.7612230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:40.7612903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:40.7741361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:40.7742007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:40.7744648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:40.7745323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:40.7771673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:40.7772310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:40.7776127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:40.7776814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:40.8407356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc50fnk_d 2022-08-17T12:32:40.8408458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc50fnk_d/_remote_module_non_scriptable.py 2022-08-17T12:32:40.9190607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk9_4cr2 2022-08-17T12:32:40.9191530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk9_4cr2/_remote_module_non_scriptable.py 2022-08-17T12:32:40.9334023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0zgsba9 2022-08-17T12:32:40.9335324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0zgsba9/_remote_module_non_scriptable.py 2022-08-17T12:32:40.9361124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgo6p496 2022-08-17T12:32:40.9363120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgo6p496/_remote_module_non_scriptable.py 2022-08-17T12:32:41.1053070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:41.2035510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:41.2036164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:41.2200326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:41.5434565Z skip: Need at least 1 CUDA device (2.177s) 2022-08-17T12:32:41.5434860Z 2022-08-17T12:32:41.5435296Z ---------------------------------------------------------------------- 2022-08-17T12:32:41.5435737Z Ran 1 test in 2.177s 2022-08-17T12:32:41.5435939Z 2022-08-17T12:32:41.5436056Z OK (skipped=1) 2022-08-17T12:32:41.5436255Z 2022-08-17T12:32:41.5436404Z Generating XML reports... 2022-08-17T12:32:41.5473113Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123239.xml 2022-08-17T12:32:42.8365303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:42.8365748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:42.8368488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:42.8368871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:42.9870906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y16jpxs 2022-08-17T12:32:42.9874577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y16jpxs/_remote_module_non_scriptable.py 2022-08-17T12:32:43.2455698Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:43.2467006Z 2022-08-17T12:32:43.2467093Z Running tests... 2022-08-17T12:32:43.2468185Z ---------------------------------------------------------------------- 2022-08-17T12:32:43.5110809Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28658 2022-08-17T12:32:43.5138829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28659 2022-08-17T12:32:43.5167587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28660 2022-08-17T12:32:43.5198470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28661 2022-08-17T12:32:44.5781243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:44.5781850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:44.5784387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:44.5785369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:44.5837336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:44.5837994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:44.5841200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:44.5841877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:44.5926579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:44.5927243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:44.5930887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:44.5931540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:44.6138864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:44.6139559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:44.6142233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:44.6143182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:44.7402116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpp8ynvy 2022-08-17T12:32:44.7402945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpp8ynvy/_remote_module_non_scriptable.py 2022-08-17T12:32:44.7441099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nkhg620 2022-08-17T12:32:44.7442284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nkhg620/_remote_module_non_scriptable.py 2022-08-17T12:32:44.7532894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3hucxhx 2022-08-17T12:32:44.7534381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3hucxhx/_remote_module_non_scriptable.py 2022-08-17T12:32:44.7734931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp563u4u5y 2022-08-17T12:32:44.7736343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp563u4u5y/_remote_module_non_scriptable.py 2022-08-17T12:32:45.0062400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:45.0063063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:45.0173295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:45.0382590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:45.3249013Z skip: Need at least 4 CUDA devices (2.078s) 2022-08-17T12:32:45.3249324Z 2022-08-17T12:32:45.3249637Z ---------------------------------------------------------------------- 2022-08-17T12:32:45.3249925Z Ran 1 test in 2.078s 2022-08-17T12:32:45.3250047Z 2022-08-17T12:32:45.3250126Z OK (skipped=1) 2022-08-17T12:32:45.3250239Z 2022-08-17T12:32:45.3250334Z Generating XML reports... 2022-08-17T12:32:45.3287616Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123243.xml 2022-08-17T12:32:46.6140425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:46.6140814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:46.6143084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:46.6143555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:46.7634941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpnxy7_z 2022-08-17T12:32:46.7637733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpnxy7_z/_remote_module_non_scriptable.py 2022-08-17T12:32:47.0210479Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:47.0221944Z 2022-08-17T12:32:47.0222322Z Running tests... 2022-08-17T12:32:47.0222717Z ---------------------------------------------------------------------- 2022-08-17T12:32:47.2849565Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28713 2022-08-17T12:32:47.2876869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28714 2022-08-17T12:32:47.2904521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28715 2022-08-17T12:32:47.2935920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28716 2022-08-17T12:32:48.3985837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:48.3986485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:48.3989653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:48.3990367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:48.4396424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:48.4397013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:48.4399417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:48.4400028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:48.4415361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:48.4416024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:48.4419408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:48.4420001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:48.4567721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:48.4568392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:48.4571577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:48.4572224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:48.5599242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rc027aj 2022-08-17T12:32:48.5600167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rc027aj/_remote_module_non_scriptable.py 2022-08-17T12:32:48.6002507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eecmn96 2022-08-17T12:32:48.6003718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eecmn96/_remote_module_non_scriptable.py 2022-08-17T12:32:48.6006686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbgl8lkk 2022-08-17T12:32:48.6009287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbgl8lkk/_remote_module_non_scriptable.py 2022-08-17T12:32:48.6151804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz50gfae 2022-08-17T12:32:48.6152952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz50gfae/_remote_module_non_scriptable.py 2022-08-17T12:32:48.8232973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:48.8606203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:48.8646111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:48.8778144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:49.1988044Z skip: Need at least 4 CUDA devices (2.176s) 2022-08-17T12:32:49.1988397Z 2022-08-17T12:32:49.1988886Z ---------------------------------------------------------------------- 2022-08-17T12:32:49.1989151Z Ran 1 test in 2.177s 2022-08-17T12:32:49.1989258Z 2022-08-17T12:32:49.1989335Z OK (skipped=1) 2022-08-17T12:32:49.1989448Z 2022-08-17T12:32:49.1989537Z Generating XML reports... 2022-08-17T12:32:49.2028122Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123247.xml 2022-08-17T12:32:50.5031494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:50.5031908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:50.5034474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:50.5035154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:50.6537709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuxnc36_ 2022-08-17T12:32:50.6540762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuxnc36_/_remote_module_non_scriptable.py 2022-08-17T12:32:50.9125687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-08-17T12:32:50.9136576Z 2022-08-17T12:32:50.9136738Z Running tests... 2022-08-17T12:32:50.9137463Z ---------------------------------------------------------------------- 2022-08-17T12:32:51.1783214Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28768 2022-08-17T12:32:51.1811521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28769 2022-08-17T12:32:51.1839951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28770 2022-08-17T12:32:51.1871085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28771 2022-08-17T12:32:52.2512338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:52.2513007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:52.2515248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:52.2516159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:52.3397153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:52.3397835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:52.3400374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:52.3401059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:52.3500347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:52.3501013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:52.3504530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:52.3505186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:52.3750255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:52.3751216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:52.3753977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:52.3754595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:52.4093500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ih96hut 2022-08-17T12:32:52.4094323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ih96hut/_remote_module_non_scriptable.py 2022-08-17T12:32:52.4996573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1wyw4dz 2022-08-17T12:32:52.4998282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1wyw4dz/_remote_module_non_scriptable.py 2022-08-17T12:32:52.5085422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mlhr0dn 2022-08-17T12:32:52.5087113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mlhr0dn/_remote_module_non_scriptable.py 2022-08-17T12:32:52.5356678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnun8ezs9 2022-08-17T12:32:52.5358072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnun8ezs9/_remote_module_non_scriptable.py 2022-08-17T12:32:52.6964023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:52.7655398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:52.7700020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:52.8048523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:53.0923020Z skip: Need at least 4 CUDA devices (2.178s) 2022-08-17T12:32:53.0923305Z 2022-08-17T12:32:53.0923769Z ---------------------------------------------------------------------- 2022-08-17T12:32:53.0924192Z Ran 1 test in 2.179s 2022-08-17T12:32:53.0924391Z 2022-08-17T12:32:53.0924493Z OK (skipped=1) 2022-08-17T12:32:53.0924673Z 2022-08-17T12:32:53.0924809Z Generating XML reports... 2022-08-17T12:32:53.0962991Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123250.xml 2022-08-17T12:32:53.7030518Z Running distributed/rpc/test_faulty_agent ... [2022-08-17 12:32:53.702646] 2022-08-17T12:32:53.7031134Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:32:53.702736] 2022-08-17T12:32:54.8152620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g0_r546 2022-08-17T12:32:54.8153747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g0_r546/_remote_module_non_scriptable.py 2022-08-17T12:32:55.0720583Z , <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]> 2022-08-17T12:32:55.0721557Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) 2022-08-17T12:32:55.0722269Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) 2022-08-17T12:32:55.0725571Z , <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]> 2022-08-17T12:32:55.0728295Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0728733Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0729080Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0729397Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0729694Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0730001Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0730328Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0730684Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0731021Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0731366Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0731714Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0732052Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0732386Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0732692Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0732981Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0733300Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0733646Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0733995Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0734331Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0734670Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) 2022-08-17T12:32:55.0735559Z , <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]> 2022-08-17T12:32:55.0736409Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:55.0736743Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:55.0737068Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:55.0737457Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:55.0737776Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:55.0738111Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) 2022-08-17T12:32:56.0199693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:56.0200083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:56.0202541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:56.0203059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:56.1732077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2allb4te 2022-08-17T12:32:56.1732706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2allb4te/_remote_module_non_scriptable.py 2022-08-17T12:32:56.4368665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:32:56.4380872Z 2022-08-17T12:32:56.4381004Z Running tests... 2022-08-17T12:32:56.4381609Z ---------------------------------------------------------------------- 2022-08-17T12:32:56.7043656Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28833 2022-08-17T12:32:56.7072773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28834 2022-08-17T12:32:56.7104424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28835 2022-08-17T12:32:56.7135032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28836 2022-08-17T12:32:57.7625794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:57.7626532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:57.7630581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:57.7631293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:57.7634970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:57.7635628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:57.7639828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:57.7640512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:57.8410280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:57.8411039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:57.8414622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:57.8415258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:57.8560995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:32:57.8561653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:32:57.8564775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:32:57.8565424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:32:57.9215766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseh5pzpf 2022-08-17T12:32:57.9216973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseh5pzpf/_remote_module_non_scriptable.py 2022-08-17T12:32:57.9233670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzhrk40f 2022-08-17T12:32:57.9236077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzhrk40f/_remote_module_non_scriptable.py 2022-08-17T12:32:57.9971187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp3mv4pz 2022-08-17T12:32:57.9972541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp3mv4pz/_remote_module_non_scriptable.py 2022-08-17T12:32:58.0124891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi92sil03 2022-08-17T12:32:58.0126224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi92sil03/_remote_module_non_scriptable.py 2022-08-17T12:32:58.1830943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:32:58.1862522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:32:58.2547890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:32:58.2705721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:32:58.3425708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:32:58.3426735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:32:58.3427466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:32:58.3428386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:32:58.3429725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:32:58.3430987Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:32:58.3432209Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:32:58.3433420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:33:05.8300915Z ok (9.392s) 2022-08-17T12:33:05.8301160Z 2022-08-17T12:33:05.8301509Z ---------------------------------------------------------------------- 2022-08-17T12:33:05.8301847Z Ran 1 test in 9.392s 2022-08-17T12:33:05.8301968Z 2022-08-17T12:33:05.8302031Z OK 2022-08-17T12:33:05.8302128Z 2022-08-17T12:33:05.8302222Z Generating XML reports... 2022-08-17T12:33:05.8337921Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220817123256.xml 2022-08-17T12:33:07.1213819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:07.1214240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:07.1215773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:07.1216151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:07.2768016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpreeu2_kr 2022-08-17T12:33:07.2768875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpreeu2_kr/_remote_module_non_scriptable.py 2022-08-17T12:33:07.5414980Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:33:07.5426114Z 2022-08-17T12:33:07.5426271Z Running tests... 2022-08-17T12:33:07.5426914Z ---------------------------------------------------------------------- 2022-08-17T12:33:07.8081920Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29020 2022-08-17T12:33:07.8111653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29021 2022-08-17T12:33:07.8140209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29022 2022-08-17T12:33:07.8171910Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29023 2022-08-17T12:33:08.8239245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:08.8239955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:08.8242079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:08.8242724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:08.8386155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:08.8386899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:08.8391142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:08.8392078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:08.8522159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:08.8522862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:08.8525484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:08.8526359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:08.8606140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:08.8606804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:08.8611005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:08.8611692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:08.9814105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb6_m5t5 2022-08-17T12:33:08.9814951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb6_m5t5/_remote_module_non_scriptable.py 2022-08-17T12:33:08.9982755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemzgm892 2022-08-17T12:33:08.9983594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemzgm892/_remote_module_non_scriptable.py 2022-08-17T12:33:09.0133808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9556py00 2022-08-17T12:33:09.0134788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9556py00/_remote_module_non_scriptable.py 2022-08-17T12:33:09.0202047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp_4uu_3 2022-08-17T12:33:09.0203758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp_4uu_3/_remote_module_non_scriptable.py 2022-08-17T12:33:09.2435957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:33:09.2611842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:33:09.2785564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:33:09.2829747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:33:15.7319448Z ok (8.189s) 2022-08-17T12:33:15.7319715Z 2022-08-17T12:33:15.7320192Z ---------------------------------------------------------------------- 2022-08-17T12:33:15.7320459Z Ran 1 test in 8.189s 2022-08-17T12:33:15.7320581Z 2022-08-17T12:33:15.7320647Z OK 2022-08-17T12:33:15.7320746Z 2022-08-17T12:33:15.7320846Z Generating XML reports... 2022-08-17T12:33:15.7355420Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220817123307.xml 2022-08-17T12:33:17.0348552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:17.0348969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:17.0351602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:17.0351983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:17.1878490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxlo8za8 2022-08-17T12:33:17.1879381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxlo8za8/_remote_module_non_scriptable.py 2022-08-17T12:33:17.4430232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:33:17.4441316Z 2022-08-17T12:33:17.4441806Z Running tests... 2022-08-17T12:33:17.4442232Z ---------------------------------------------------------------------- 2022-08-17T12:33:17.7054712Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29195 2022-08-17T12:33:17.7082804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29196 2022-08-17T12:33:17.7111190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29197 2022-08-17T12:33:17.7142002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29198 2022-08-17T12:33:18.6970589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:18.6971261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:18.6973689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:18.6974368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:18.7394524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:18.7395172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:18.7398397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:18.7399076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:18.7542064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:18.7542716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:18.7545426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:18.7546064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:18.7630836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:18.7631513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:18.7634735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:18.7635431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:18.8585408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy7f9r38 2022-08-17T12:33:18.8586165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy7f9r38/_remote_module_non_scriptable.py 2022-08-17T12:33:18.9017329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkc7tj16 2022-08-17T12:33:18.9018775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkc7tj16/_remote_module_non_scriptable.py 2022-08-17T12:33:18.9167996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pnlzhfg 2022-08-17T12:33:18.9168769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pnlzhfg/_remote_module_non_scriptable.py 2022-08-17T12:33:18.9214259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3igbasn 2022-08-17T12:33:18.9216408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3igbasn/_remote_module_non_scriptable.py 2022-08-17T12:33:19.1240680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:33:19.1685022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:33:19.1824865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:33:19.1835686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:33:25.6287928Z ok (8.184s) 2022-08-17T12:33:25.6288165Z 2022-08-17T12:33:25.6288650Z ---------------------------------------------------------------------- 2022-08-17T12:33:25.6288932Z Ran 1 test in 8.185s 2022-08-17T12:33:25.6289056Z 2022-08-17T12:33:25.6289127Z OK 2022-08-17T12:33:25.6289611Z 2022-08-17T12:33:25.6289793Z Generating XML reports... 2022-08-17T12:33:25.6325370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123317.xml 2022-08-17T12:33:26.9443907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:26.9444297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:26.9446996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:26.9447382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:27.0992360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyangfce 2022-08-17T12:33:27.0993106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyangfce/_remote_module_non_scriptable.py 2022-08-17T12:33:27.3589319Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:33:27.3600515Z 2022-08-17T12:33:27.3600710Z Running tests... 2022-08-17T12:33:27.3601212Z ---------------------------------------------------------------------- 2022-08-17T12:33:27.6274161Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29370 2022-08-17T12:33:27.6302820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29371 2022-08-17T12:33:27.6332000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29372 2022-08-17T12:33:27.6362378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29373 2022-08-17T12:33:28.7006293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:28.7006987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:28.7009242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:28.7009881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:28.7136680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:28.7137315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:28.7139942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:28.7140618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:28.7301457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:28.7302035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:28.7304808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:28.7305427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:28.7306481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:28.7307056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:28.7311831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:28.7312624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:28.8639029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqurfpi5q 2022-08-17T12:33:28.8640042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqurfpi5q/_remote_module_non_scriptable.py 2022-08-17T12:33:28.8767656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehnj8wsn 2022-08-17T12:33:28.8768490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehnj8wsn/_remote_module_non_scriptable.py 2022-08-17T12:33:28.8957198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub7j9slm 2022-08-17T12:33:28.8958095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub7j9slm/_remote_module_non_scriptable.py 2022-08-17T12:33:28.9095393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8378ebx 2022-08-17T12:33:28.9096335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8378ebx/_remote_module_non_scriptable.py 2022-08-17T12:33:29.1246063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:33:29.1367601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:33:29.1530224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:33:29.1703421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:33:35.5504500Z ok (8.190s) 2022-08-17T12:33:35.5504831Z 2022-08-17T12:33:35.5505329Z ---------------------------------------------------------------------- 2022-08-17T12:33:35.5505603Z Ran 1 test in 8.190s 2022-08-17T12:33:35.5505800Z 2022-08-17T12:33:35.5505866Z OK 2022-08-17T12:33:35.5505949Z 2022-08-17T12:33:35.5506050Z Generating XML reports... 2022-08-17T12:33:35.5541456Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123327.xml 2022-08-17T12:33:36.8549631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:36.8550035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:36.8552622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:36.8553031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:37.0077024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7j1q3csa 2022-08-17T12:33:37.0077688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7j1q3csa/_remote_module_non_scriptable.py 2022-08-17T12:33:37.2642008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:33:37.2652911Z 2022-08-17T12:33:37.2653059Z Running tests... 2022-08-17T12:33:37.2654267Z ---------------------------------------------------------------------- 2022-08-17T12:33:37.5314441Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29545 2022-08-17T12:33:37.5343785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29546 2022-08-17T12:33:37.5374102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29547 2022-08-17T12:33:37.5412366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29548 2022-08-17T12:33:38.6091028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:38.6091721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:38.6092913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:38.6093526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:38.6126013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:38.6126703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:38.6130237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:38.6131158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:38.6189531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:38.6190206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:38.6194549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:38.6195220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:38.6232403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:38.6233103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:38.6237093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:38.6237866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:38.7691563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54z4oeqx 2022-08-17T12:33:38.7692543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54z4oeqx/_remote_module_non_scriptable.py 2022-08-17T12:33:38.7726387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmv25x9w 2022-08-17T12:33:38.7727831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmv25x9w/_remote_module_non_scriptable.py 2022-08-17T12:33:38.7774019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hovrwu8 2022-08-17T12:33:38.7776222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hovrwu8/_remote_module_non_scriptable.py 2022-08-17T12:33:38.7803256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptse8na0 2022-08-17T12:33:38.7804928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptse8na0/_remote_module_non_scriptable.py 2022-08-17T12:33:39.0269488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:33:39.0317787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:33:39.0363615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:33:39.0364182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:33:50.0627746Z ok (12.797s) 2022-08-17T12:33:50.0628006Z 2022-08-17T12:33:50.0628499Z ---------------------------------------------------------------------- 2022-08-17T12:33:50.0629181Z Ran 1 test in 12.797s 2022-08-17T12:33:50.0629364Z 2022-08-17T12:33:50.0629439Z OK 2022-08-17T12:33:50.0629588Z 2022-08-17T12:33:50.0629731Z Generating XML reports... 2022-08-17T12:33:50.0665727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123337.xml 2022-08-17T12:33:51.3588036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:51.3588452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:51.3590793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:51.3591267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:51.5146402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcw7kc0vm 2022-08-17T12:33:51.5147501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcw7kc0vm/_remote_module_non_scriptable.py 2022-08-17T12:33:51.7760120Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:33:51.7771104Z 2022-08-17T12:33:51.7771203Z Running tests... 2022-08-17T12:33:51.7772621Z ---------------------------------------------------------------------- 2022-08-17T12:33:52.0414734Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29720 2022-08-17T12:33:52.0442745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29721 2022-08-17T12:33:52.0471837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29722 2022-08-17T12:33:52.0502737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29723 2022-08-17T12:33:53.0760329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:53.0761015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:53.0763700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:53.0764396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:53.0781366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:53.0781992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:53.0785493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:53.0786172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:53.1315543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:53.1316180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:53.1319751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:53.1330843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:53.1331870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:33:53.1332428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:33:53.1333218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:33:53.1333830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:33:53.2343423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79ab42dr 2022-08-17T12:33:53.2344326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79ab42dr/_remote_module_non_scriptable.py 2022-08-17T12:33:53.2363423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnjgp65n 2022-08-17T12:33:53.2365122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnjgp65n/_remote_module_non_scriptable.py 2022-08-17T12:33:53.2916147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkpkvf5v 2022-08-17T12:33:53.2916938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkpkvf5v/_remote_module_non_scriptable.py 2022-08-17T12:33:53.2924110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsn92cmo 2022-08-17T12:33:53.2926477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsn92cmo/_remote_module_non_scriptable.py 2022-08-17T12:33:53.4931331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:33:53.4963874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:33:53.5541988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:33:53.5557334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:33:59.9650697Z ok (8.188s) 2022-08-17T12:33:59.9650866Z 2022-08-17T12:33:59.9651458Z ---------------------------------------------------------------------- 2022-08-17T12:33:59.9651827Z Ran 1 test in 8.188s 2022-08-17T12:33:59.9651935Z 2022-08-17T12:33:59.9652000Z OK 2022-08-17T12:33:59.9652097Z 2022-08-17T12:33:59.9652196Z Generating XML reports... 2022-08-17T12:33:59.9687987Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123351.xml 2022-08-17T12:34:01.2687096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:01.2687492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:01.2690262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:01.2690644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:01.4233577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1iw4df2 2022-08-17T12:34:01.4234794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1iw4df2/_remote_module_non_scriptable.py 2022-08-17T12:34:01.6838657Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:01.6850091Z 2022-08-17T12:34:01.6850453Z Running tests... 2022-08-17T12:34:01.6851090Z ---------------------------------------------------------------------- 2022-08-17T12:34:01.9533369Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29895 2022-08-17T12:34:01.9562513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29896 2022-08-17T12:34:01.9592028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29897 2022-08-17T12:34:01.9623696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29898 2022-08-17T12:34:03.0762568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:03.0765053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:03.0810299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:03.0816360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:03.1017504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:03.1018155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:03.1021359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:03.1022296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:03.1275624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:03.1276245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:03.1279266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:03.1279858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:03.1375805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:03.1376463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:03.1380770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:03.1381446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:03.2402886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcc91z07 2022-08-17T12:34:03.2404209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcc91z07/_remote_module_non_scriptable.py 2022-08-17T12:34:03.2678542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxry85g51 2022-08-17T12:34:03.2679338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxry85g51/_remote_module_non_scriptable.py 2022-08-17T12:34:03.2881753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi43_rwm 2022-08-17T12:34:03.2882502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi43_rwm/_remote_module_non_scriptable.py 2022-08-17T12:34:03.2997955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcndsam6 2022-08-17T12:34:03.2998981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcndsam6/_remote_module_non_scriptable.py 2022-08-17T12:34:03.5041691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:03.5310579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:03.5494198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:03.5627561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:04.0678540Z ok (2.383s) 2022-08-17T12:34:04.0678712Z 2022-08-17T12:34:04.0679146Z ---------------------------------------------------------------------- 2022-08-17T12:34:04.0679430Z Ran 1 test in 2.383s 2022-08-17T12:34:04.0679555Z 2022-08-17T12:34:04.0679605Z OK 2022-08-17T12:34:04.0679702Z 2022-08-17T12:34:04.0679797Z Generating XML reports... 2022-08-17T12:34:04.0715659Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123401.xml 2022-08-17T12:34:05.3801147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:05.3801841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:05.3804440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:05.3805052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:05.5339931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95yffqae 2022-08-17T12:34:05.5341844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95yffqae/_remote_module_non_scriptable.py 2022-08-17T12:34:05.7955970Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:05.7968171Z 2022-08-17T12:34:05.7968578Z Running tests... 2022-08-17T12:34:05.7969252Z ---------------------------------------------------------------------- 2022-08-17T12:34:06.0601416Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30070 2022-08-17T12:34:06.0630349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30071 2022-08-17T12:34:06.0659064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30072 2022-08-17T12:34:06.0691084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30073 2022-08-17T12:34:07.1662964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:07.1663631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:07.1666871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:07.1667563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:07.1969262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:07.1969921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:07.1972563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:07.1973254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:07.2079008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:07.2079701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:07.2082635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:07.2083337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:07.2528041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:07.2528599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:07.2531559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:07.2532182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:07.3508792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmjjo3x7 2022-08-17T12:34:07.3509607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmjjo3x7/_remote_module_non_scriptable.py 2022-08-17T12:34:07.3593734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0mcmij8 2022-08-17T12:34:07.3594526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0mcmij8/_remote_module_non_scriptable.py 2022-08-17T12:34:07.3689273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprne6joyl 2022-08-17T12:34:07.3690294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprne6joyl/_remote_module_non_scriptable.py 2022-08-17T12:34:07.4136708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu16yxor 2022-08-17T12:34:07.4137724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu16yxor/_remote_module_non_scriptable.py 2022-08-17T12:34:07.6127078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:07.6201135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:07.6293655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:07.6750708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:14.0839584Z ok (8.287s) 2022-08-17T12:34:14.0839862Z 2022-08-17T12:34:14.0840308Z ---------------------------------------------------------------------- 2022-08-17T12:34:14.0840559Z Ran 1 test in 8.287s 2022-08-17T12:34:14.0840680Z 2022-08-17T12:34:14.0840743Z OK 2022-08-17T12:34:14.0840838Z 2022-08-17T12:34:14.0840933Z Generating XML reports... 2022-08-17T12:34:14.0876464Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123405.xml 2022-08-17T12:34:15.3905739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:15.3906135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:15.3909304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:15.3909706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:15.5425262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx21pplk 2022-08-17T12:34:15.5426922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx21pplk/_remote_module_non_scriptable.py 2022-08-17T12:34:15.7993787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:15.8004859Z 2022-08-17T12:34:15.8005017Z Running tests... 2022-08-17T12:34:15.8005598Z ---------------------------------------------------------------------- 2022-08-17T12:34:16.0646334Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30245 2022-08-17T12:34:16.0674307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30246 2022-08-17T12:34:16.0703593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30247 2022-08-17T12:34:16.0735550Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30248 2022-08-17T12:34:17.0855527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:17.0856154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:17.0858646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:17.0859333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:17.0921867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:17.0922435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:17.0926410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:17.0927083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:17.1006118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:17.1006803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:17.1010912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:17.1011584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:17.1148608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:17.1149272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:17.1152019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:17.1152679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:17.2469373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ti6wuue 2022-08-17T12:34:17.2470514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ti6wuue/_remote_module_non_scriptable.py 2022-08-17T12:34:17.2516610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8epxb0w5 2022-08-17T12:34:17.2518038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8epxb0w5/_remote_module_non_scriptable.py 2022-08-17T12:34:17.2611941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnadk4yf 2022-08-17T12:34:17.2613238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnadk4yf/_remote_module_non_scriptable.py 2022-08-17T12:34:17.2745055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6siw8tr 2022-08-17T12:34:17.2746025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6siw8tr/_remote_module_non_scriptable.py 2022-08-17T12:34:17.5085619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:17.5122671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:17.5207560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:17.5357165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:22.9867335Z ok (7.186s) 2022-08-17T12:34:22.9867649Z 2022-08-17T12:34:22.9868187Z ---------------------------------------------------------------------- 2022-08-17T12:34:22.9868648Z Ran 1 test in 7.186s 2022-08-17T12:34:22.9868762Z 2022-08-17T12:34:22.9868829Z OK 2022-08-17T12:34:22.9868926Z 2022-08-17T12:34:22.9869023Z Generating XML reports... 2022-08-17T12:34:22.9903871Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123415.xml 2022-08-17T12:34:24.2859327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:24.2859739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:24.2861775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:24.2862327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:24.4386925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbo3c_2wn 2022-08-17T12:34:24.4387881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbo3c_2wn/_remote_module_non_scriptable.py 2022-08-17T12:34:24.6949550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:24.6960138Z 2022-08-17T12:34:24.6960259Z Running tests... 2022-08-17T12:34:24.6960760Z ---------------------------------------------------------------------- 2022-08-17T12:34:24.9591146Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30420 2022-08-17T12:34:24.9619056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30421 2022-08-17T12:34:24.9647128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30422 2022-08-17T12:34:24.9677773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30423 2022-08-17T12:34:26.0301893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:26.0302591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:26.0305317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:26.0305971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:26.0577551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:26.0580796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:26.0581613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:26.0582258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:26.0583651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:26.0584307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:26.0588784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:26.0589425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:26.1066528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:26.1067297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:26.1069670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:26.1070392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:26.1893475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbj1cuoh 2022-08-17T12:34:26.1894375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbj1cuoh/_remote_module_non_scriptable.py 2022-08-17T12:34:26.2185571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51tkgr15 2022-08-17T12:34:26.2186641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51tkgr15/_remote_module_non_scriptable.py 2022-08-17T12:34:26.2206883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4999z8p 2022-08-17T12:34:26.2209828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4999z8p/_remote_module_non_scriptable.py 2022-08-17T12:34:26.2656639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rj_rhrs 2022-08-17T12:34:26.2657439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rj_rhrs/_remote_module_non_scriptable.py 2022-08-17T12:34:26.4491839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:26.4800661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:26.4836685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:26.5260068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:28.6073865Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-08-17T12:34:28.6075723Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first): 2022-08-17T12:34:28.6077502Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7fd9aa506c5b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:34:28.6079288Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7fd9aa50262e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:34:28.6081218Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7fd9aa5043ce in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:34:28.6083324Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7fd9aea6a170 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6085744Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7fd9aea5a74c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6088456Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7fd9b6a04b0a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:34:28.6090869Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7fd9aea5ee9c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6093225Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7fd9b6a00b15 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:34:28.6098190Z frame #8: + 0x4511353 (0x7fd9aea58353 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6099976Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7fd9aea58f48 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6103052Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7fd9aea535e7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6104468Z frame #11: + 0x4541232 (0x7fd9aea88232 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:34:28.6105668Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fd9aa4f508b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:34:28.6106529Z frame #13: + 0xdbbf4 (0x7fd9aa383bf4 in /opt/conda/lib/libstdc++.so.6) 2022-08-17T12:34:28.6107528Z frame #14: + 0x8609 (0x7fd9be891609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-08-17T12:34:28.6108406Z frame #15: clone + 0x43 (0x7fd9be7b6133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-08-17T12:34:28.6108771Z 2022-08-17T12:34:28.9763941Z ok (4.280s) 2022-08-17T12:34:28.9764187Z 2022-08-17T12:34:28.9764734Z ---------------------------------------------------------------------- 2022-08-17T12:34:28.9765040Z Ran 1 test in 4.280s 2022-08-17T12:34:28.9765186Z 2022-08-17T12:34:28.9765252Z OK 2022-08-17T12:34:28.9765350Z 2022-08-17T12:34:28.9765447Z Generating XML reports... 2022-08-17T12:34:28.9800423Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123424.xml 2022-08-17T12:34:30.2974833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:30.2975239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:30.2977770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:30.2978403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:30.4503874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dsahpdq 2022-08-17T12:34:30.4504848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dsahpdq/_remote_module_non_scriptable.py 2022-08-17T12:34:30.7079359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:30.7090643Z 2022-08-17T12:34:30.7090793Z Running tests... 2022-08-17T12:34:30.7091576Z ---------------------------------------------------------------------- 2022-08-17T12:34:30.9739553Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30595 2022-08-17T12:34:30.9769098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30596 2022-08-17T12:34:30.9798332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30597 2022-08-17T12:34:30.9830711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30598 2022-08-17T12:34:32.0089638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:32.0090685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:32.0092904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:32.0093555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:32.0221836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:32.0222505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:32.0225198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:32.0225886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:32.0351251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:32.0351892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:32.0355291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:32.0355933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:32.0386610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:32.0387220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:32.0391750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:32.0392402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:32.1710996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphulu2tq5 2022-08-17T12:34:32.1711756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphulu2tq5/_remote_module_non_scriptable.py 2022-08-17T12:34:32.1847352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7w631sgm 2022-08-17T12:34:32.1848997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7w631sgm/_remote_module_non_scriptable.py 2022-08-17T12:34:32.1949698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kxbyfah 2022-08-17T12:34:32.1950520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kxbyfah/_remote_module_non_scriptable.py 2022-08-17T12:34:32.1983020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxxwldy3 2022-08-17T12:34:32.1984156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxxwldy3/_remote_module_non_scriptable.py 2022-08-17T12:34:32.4346171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:32.4473037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:32.4554997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:32.4587279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:38.8972786Z ok (8.188s) 2022-08-17T12:34:38.8973002Z 2022-08-17T12:34:38.8973372Z ---------------------------------------------------------------------- 2022-08-17T12:34:38.8973733Z Ran 1 test in 8.188s 2022-08-17T12:34:38.8973859Z 2022-08-17T12:34:38.8973930Z OK 2022-08-17T12:34:38.8974026Z 2022-08-17T12:34:38.8974123Z Generating XML reports... 2022-08-17T12:34:38.9010714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123430.xml 2022-08-17T12:34:40.2066767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:40.2067173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:40.2069904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:40.2070293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:40.3590057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi69_0gk 2022-08-17T12:34:40.3590989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi69_0gk/_remote_module_non_scriptable.py 2022-08-17T12:34:40.6193591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:40.6204530Z 2022-08-17T12:34:40.6204845Z Running tests... 2022-08-17T12:34:40.6205528Z ---------------------------------------------------------------------- 2022-08-17T12:34:40.8828097Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30770 2022-08-17T12:34:40.8856882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30771 2022-08-17T12:34:40.8885251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30772 2022-08-17T12:34:40.8918825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30773 2022-08-17T12:34:41.8873104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:41.8873773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:41.8876457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:41.8877109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:41.8969336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:41.8970024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:41.8972460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:41.8973134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:41.9852418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:41.9853043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:41.9856001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:41.9856682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:41.9939836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:41.9940522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:41.9943581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:41.9944177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:42.0500692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuonjvme2 2022-08-17T12:34:42.0501667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuonjvme2/_remote_module_non_scriptable.py 2022-08-17T12:34:42.0600175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwigdn30s 2022-08-17T12:34:42.0601215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwigdn30s/_remote_module_non_scriptable.py 2022-08-17T12:34:42.1458149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hpz93m6 2022-08-17T12:34:42.1458986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hpz93m6/_remote_module_non_scriptable.py 2022-08-17T12:34:42.1542478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg31v08dv 2022-08-17T12:34:42.1543767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg31v08dv/_remote_module_non_scriptable.py 2022-08-17T12:34:42.3117127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:42.3238256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:42.4105387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:42.4147847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:48.8063856Z ok (8.186s) 2022-08-17T12:34:48.8064121Z 2022-08-17T12:34:48.8064525Z ---------------------------------------------------------------------- 2022-08-17T12:34:48.8064805Z Ran 1 test in 8.186s 2022-08-17T12:34:48.8064928Z 2022-08-17T12:34:48.8064994Z OK 2022-08-17T12:34:48.8065077Z 2022-08-17T12:34:48.8065173Z Generating XML reports... 2022-08-17T12:34:48.8101295Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123440.xml 2022-08-17T12:34:50.1288820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:50.1289218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:50.1291820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:50.1292492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:50.2812257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe8yc6a1 2022-08-17T12:34:50.2813079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe8yc6a1/_remote_module_non_scriptable.py 2022-08-17T12:34:50.5394002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:50.5404846Z 2022-08-17T12:34:50.5405000Z Running tests... 2022-08-17T12:34:50.5405563Z ---------------------------------------------------------------------- 2022-08-17T12:34:50.8027108Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30945 2022-08-17T12:34:50.8057248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30946 2022-08-17T12:34:50.8086510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30947 2022-08-17T12:34:50.8117808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30948 2022-08-17T12:34:51.8728129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:51.8729046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:51.8732057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:51.8732683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:51.8788224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:51.8788903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:51.8789743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:51.8790269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:51.8792497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:51.8793108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:51.8793867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:51.8794772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:51.8846196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:51.8846837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:51.8851257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:51.8851945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:52.0367969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_m9vz5_ 2022-08-17T12:34:52.0368720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_m9vz5_/_remote_module_non_scriptable.py 2022-08-17T12:34:52.0412696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwnjt2br 2022-08-17T12:34:52.0414256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwnjt2br/_remote_module_non_scriptable.py 2022-08-17T12:34:52.0415003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkot42f05 2022-08-17T12:34:52.0417175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkot42f05/_remote_module_non_scriptable.py 2022-08-17T12:34:52.0450210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx15feskh 2022-08-17T12:34:52.0451731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx15feskh/_remote_module_non_scriptable.py 2022-08-17T12:34:52.3018620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:34:52.3048222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:34:52.3048925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:34:52.3292178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:34:57.8249150Z ok (7.284s) 2022-08-17T12:34:57.8249385Z 2022-08-17T12:34:57.8249710Z ---------------------------------------------------------------------- 2022-08-17T12:34:57.8249974Z Ran 1 test in 7.284s 2022-08-17T12:34:57.8250100Z 2022-08-17T12:34:57.8250165Z OK 2022-08-17T12:34:57.8250261Z 2022-08-17T12:34:57.8250342Z Generating XML reports... 2022-08-17T12:34:57.8286505Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123450.xml 2022-08-17T12:34:59.1373227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:34:59.1373623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:34:59.1376072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:34:59.1376571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:34:59.2881839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz49y538i 2022-08-17T12:34:59.2882276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz49y538i/_remote_module_non_scriptable.py 2022-08-17T12:34:59.5463053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:34:59.5474275Z 2022-08-17T12:34:59.5474548Z Running tests... 2022-08-17T12:34:59.5475001Z ---------------------------------------------------------------------- 2022-08-17T12:34:59.8088920Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31120 2022-08-17T12:34:59.8119175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31121 2022-08-17T12:34:59.8148152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31122 2022-08-17T12:34:59.8179084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31123 2022-08-17T12:35:00.8240520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:00.8241244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:00.8242979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:00.8243634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:00.8669214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:00.8669863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:00.8672038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:00.8672689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:00.8773041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:00.8773724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:00.8777666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:00.8778348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:00.9103633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:00.9104309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:00.9107539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:00.9108116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:00.9810028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7ybbg9l 2022-08-17T12:35:00.9810952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7ybbg9l/_remote_module_non_scriptable.py 2022-08-17T12:35:01.0277835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4t4x3ke 2022-08-17T12:35:01.0279076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4t4x3ke/_remote_module_non_scriptable.py 2022-08-17T12:35:01.0362235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfjupq31 2022-08-17T12:35:01.0363600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfjupq31/_remote_module_non_scriptable.py 2022-08-17T12:35:01.0691212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppozqn1yh 2022-08-17T12:35:01.0692518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppozqn1yh/_remote_module_non_scriptable.py 2022-08-17T12:35:01.2396860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:01.2882511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:01.2961863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:01.3268599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:03.4179265Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-08-17T12:35:03.4180784Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first): 2022-08-17T12:35:03.4182902Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f97ed29cc5b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:03.4184762Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f97ed29862e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:03.4186720Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7f97ed29a3ce in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:03.4188580Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7f97f1800170 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4190953Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7f97f17f074c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4193332Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7f97f979ab0a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:35:03.4195612Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7f97f17f4e9c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4197983Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f97f9796b15 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:35:03.4199561Z frame #8: + 0x4511353 (0x7f97f17ee353 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4201272Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7f97f17eef48 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4203342Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f97f17e95e7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4204740Z frame #11: + 0x4541232 (0x7f97f181e232 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:03.4206125Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f97ed28b08b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:03.4206984Z frame #13: + 0xdbbf4 (0x7f97ed119bf4 in /opt/conda/lib/libstdc++.so.6) 2022-08-17T12:35:03.4207913Z frame #14: + 0x8609 (0x7f9801627609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-08-17T12:35:03.4208756Z frame #15: clone + 0x43 (0x7f980154c133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-08-17T12:35:03.4209119Z 2022-08-17T12:35:03.8268239Z ok (4.279s) 2022-08-17T12:35:03.8268490Z 2022-08-17T12:35:03.8268950Z ---------------------------------------------------------------------- 2022-08-17T12:35:03.8269347Z Ran 1 test in 4.279s 2022-08-17T12:35:03.8269545Z 2022-08-17T12:35:03.8269644Z OK 2022-08-17T12:35:03.8269814Z 2022-08-17T12:35:03.8270287Z Generating XML reports... 2022-08-17T12:35:03.8306408Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123459.xml 2022-08-17T12:35:05.1275052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:05.1275457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:05.1278282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:05.1278672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:05.2804701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yjkfeg6 2022-08-17T12:35:05.2806726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yjkfeg6/_remote_module_non_scriptable.py 2022-08-17T12:35:05.5386545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:05.5397789Z 2022-08-17T12:35:05.5397912Z Running tests... 2022-08-17T12:35:05.5398407Z ---------------------------------------------------------------------- 2022-08-17T12:35:05.8047191Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31295 2022-08-17T12:35:05.8076223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31296 2022-08-17T12:35:05.8104314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31297 2022-08-17T12:35:05.8137379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31298 2022-08-17T12:35:06.8820314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:06.8820979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:06.8824108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:06.8824711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:06.9039581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:06.9040233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:06.9043515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:06.9044149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:06.9119478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:06.9120155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:06.9124556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:06.9125217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:06.9246815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:06.9247531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:06.9250434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:06.9251117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:07.0444234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvo10fec 2022-08-17T12:35:07.0445086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvo10fec/_remote_module_non_scriptable.py 2022-08-17T12:35:07.0740475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2dm_59ng 2022-08-17T12:35:07.0742332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2dm_59ng/_remote_module_non_scriptable.py 2022-08-17T12:35:07.0863020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy72zkz95 2022-08-17T12:35:07.0863730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy72zkz95/_remote_module_non_scriptable.py 2022-08-17T12:35:07.0882419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e_ru53h 2022-08-17T12:35:07.0884498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e_ru53h/_remote_module_non_scriptable.py 2022-08-17T12:35:07.3143611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:07.3393977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:07.3492034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:07.3511438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:21.2402414Z ok (15.700s) 2022-08-17T12:35:21.2402673Z 2022-08-17T12:35:21.2403143Z ---------------------------------------------------------------------- 2022-08-17T12:35:21.2403430Z Ran 1 test in 15.700s 2022-08-17T12:35:21.2403578Z 2022-08-17T12:35:21.2403648Z OK 2022-08-17T12:35:21.2403744Z 2022-08-17T12:35:21.2403843Z Generating XML reports... 2022-08-17T12:35:21.2444442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123505.xml 2022-08-17T12:35:22.5717214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:22.5717659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:22.5720651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:22.5721095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:22.7251675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfjzpm3y 2022-08-17T12:35:22.7252764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfjzpm3y/_remote_module_non_scriptable.py 2022-08-17T12:35:22.9861540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:22.9872730Z 2022-08-17T12:35:22.9872846Z Running tests... 2022-08-17T12:35:22.9873305Z ---------------------------------------------------------------------- 2022-08-17T12:35:23.2528467Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31470 2022-08-17T12:35:23.2558317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31471 2022-08-17T12:35:23.2616069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31472 2022-08-17T12:35:23.2647268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31473 2022-08-17T12:35:24.2760424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:24.2761089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:24.2762235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:24.2762875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:24.2923301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:24.2923981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:24.2927205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:24.2928124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:24.2999745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:24.3000373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:24.3004890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:24.3005569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:24.3101069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:24.3101484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:24.3104719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:24.3105290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:24.4404380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_s1vjda 2022-08-17T12:35:24.4405212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_s1vjda/_remote_module_non_scriptable.py 2022-08-17T12:35:24.4593170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzmmzhna 2022-08-17T12:35:24.4593879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzmmzhna/_remote_module_non_scriptable.py 2022-08-17T12:35:24.4718410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_2ic07p 2022-08-17T12:35:24.4719201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_2ic07p/_remote_module_non_scriptable.py 2022-08-17T12:35:24.4728593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9sg8avf 2022-08-17T12:35:24.4730460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9sg8avf/_remote_module_non_scriptable.py 2022-08-17T12:35:24.7025332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:24.7207469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:24.7298104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:24.7298731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:32.6818269Z ok (9.694s) 2022-08-17T12:35:32.6818419Z 2022-08-17T12:35:32.6818761Z ---------------------------------------------------------------------- 2022-08-17T12:35:32.6819027Z Ran 1 test in 9.694s 2022-08-17T12:35:32.6819409Z 2022-08-17T12:35:32.6819474Z OK 2022-08-17T12:35:32.6819571Z 2022-08-17T12:35:32.6819657Z Generating XML reports... 2022-08-17T12:35:32.6857878Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123522.xml 2022-08-17T12:35:33.9893542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:33.9893925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:33.9896558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:33.9896936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:34.1446705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oh91thj 2022-08-17T12:35:34.1447606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oh91thj/_remote_module_non_scriptable.py 2022-08-17T12:35:34.4025354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:34.4036409Z 2022-08-17T12:35:34.4036545Z Running tests... 2022-08-17T12:35:34.4037608Z ---------------------------------------------------------------------- 2022-08-17T12:35:34.6681415Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31645 2022-08-17T12:35:34.6709257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31646 2022-08-17T12:35:34.6737568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31647 2022-08-17T12:35:34.6769279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31648 2022-08-17T12:35:35.7152449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:35.7153073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:35.7155329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:35.7155968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:35.7215243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:35.7215894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:35.7219610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:35.7220296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:35.7222880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:35.7223517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:35.7227655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:35.7228381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:35.7327672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:35.7328352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:35.7332522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:35.7333177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:35.8767127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf33c90g 2022-08-17T12:35:35.8768252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf33c90g/_remote_module_non_scriptable.py 2022-08-17T12:35:35.8816503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qvf_r2e 2022-08-17T12:35:35.8818002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qvf_r2e/_remote_module_non_scriptable.py 2022-08-17T12:35:35.8827967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_08wz9xb 2022-08-17T12:35:35.8830308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_08wz9xb/_remote_module_non_scriptable.py 2022-08-17T12:35:35.8951823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfxw564u 2022-08-17T12:35:35.8952787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfxw564u/_remote_module_non_scriptable.py 2022-08-17T12:35:36.1448861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:36.1463674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:36.1489530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:36.1799908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:38.6855253Z ok (4.282s) 2022-08-17T12:35:38.6855497Z 2022-08-17T12:35:38.6855826Z ---------------------------------------------------------------------- 2022-08-17T12:35:38.6856351Z Ran 1 test in 4.282s 2022-08-17T12:35:38.6856479Z 2022-08-17T12:35:38.6856545Z OK 2022-08-17T12:35:38.6856642Z 2022-08-17T12:35:38.6856739Z Generating XML reports... 2022-08-17T12:35:38.6891483Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123534.xml 2022-08-17T12:35:39.9775273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:39.9775666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:39.9778205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:39.9778601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:40.1311345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_0qroqs 2022-08-17T12:35:40.1312240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_0qroqs/_remote_module_non_scriptable.py 2022-08-17T12:35:40.3924379Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:40.3935131Z 2022-08-17T12:35:40.3935241Z Running tests... 2022-08-17T12:35:40.3935878Z ---------------------------------------------------------------------- 2022-08-17T12:35:40.6560173Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31820 2022-08-17T12:35:40.6588775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31821 2022-08-17T12:35:40.6617496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31822 2022-08-17T12:35:40.6649563Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31823 2022-08-17T12:35:41.7339566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:41.7340248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:41.7343769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:41.7344445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:41.7348756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:41.7349407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:41.7353712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:41.7354686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:41.7433763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:41.7437898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:41.7438811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:41.7439444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:41.7665431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:41.7666113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:41.7668808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:41.7669488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:41.8939904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3_5i_fr 2022-08-17T12:35:41.8941198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3_5i_fr/_remote_module_non_scriptable.py 2022-08-17T12:35:41.8961933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb0z7uz2 2022-08-17T12:35:41.8964461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb0z7uz2/_remote_module_non_scriptable.py 2022-08-17T12:35:41.9032799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ep7xdgb 2022-08-17T12:35:41.9034809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ep7xdgb/_remote_module_non_scriptable.py 2022-08-17T12:35:41.9265098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxs34894 2022-08-17T12:35:41.9265927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxs34894/_remote_module_non_scriptable.py 2022-08-17T12:35:42.1527632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:42.1574824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:42.1641148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:42.1884560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:48.5793945Z ok (8.185s) 2022-08-17T12:35:48.5794193Z 2022-08-17T12:35:48.5794683Z ---------------------------------------------------------------------- 2022-08-17T12:35:48.5795602Z Ran 1 test in 8.186s 2022-08-17T12:35:48.5795796Z 2022-08-17T12:35:48.5795869Z OK 2022-08-17T12:35:48.5795971Z 2022-08-17T12:35:48.5796058Z Generating XML reports... 2022-08-17T12:35:48.5833814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123540.xml 2022-08-17T12:35:49.8833287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:49.8833984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:49.8836822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:49.8837415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:50.0373854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn27x09b 2022-08-17T12:35:50.0374360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn27x09b/_remote_module_non_scriptable.py 2022-08-17T12:35:50.3009148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:50.3019963Z 2022-08-17T12:35:50.3020067Z Running tests... 2022-08-17T12:35:50.3020890Z ---------------------------------------------------------------------- 2022-08-17T12:35:50.5683036Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31995 2022-08-17T12:35:50.5712924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31996 2022-08-17T12:35:50.5741653Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31997 2022-08-17T12:35:50.5773435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31998 2022-08-17T12:35:51.5704440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:51.5705104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:51.5709167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:51.5709857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:51.6147573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:51.6148215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:51.6150707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:51.6151316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:51.6152186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:51.6152817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:51.6155367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:51.6155973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:51.6307599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:51.6308254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:51.6311646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:51.6312316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:51.7298361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekdaxhoz 2022-08-17T12:35:51.7299208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekdaxhoz/_remote_module_non_scriptable.py 2022-08-17T12:35:51.7745312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd_bjna5 2022-08-17T12:35:51.7746090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd_bjna5/_remote_module_non_scriptable.py 2022-08-17T12:35:51.7756718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvani4p4 2022-08-17T12:35:51.7758800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvani4p4/_remote_module_non_scriptable.py 2022-08-17T12:35:51.7898198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uj76cjj 2022-08-17T12:35:51.7899706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uj76cjj/_remote_module_non_scriptable.py 2022-08-17T12:35:51.9926418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:35:52.0341176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:52.0370894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:52.0513916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:55.1059473Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-08-17T12:35:55.1060562Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first): 2022-08-17T12:35:55.1061384Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ffa10abec5b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:55.1062187Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7ffa10aba62e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:55.1063061Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7ffa10abc3ce in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:55.1063938Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7ffa15022170 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1064962Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7ffa1501274c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1066005Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0xcc (0x7ffa1cfb94dc in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:35:55.1067078Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ffa15016ee4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1068112Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ffa1cfb8b15 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-08-17T12:35:55.1068819Z frame #8: + 0x4511353 (0x7ffa15010353 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1069568Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7ffa15010f48 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1070429Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ffa1500b5e7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1071040Z frame #11: + 0x4541232 (0x7ffa15040232 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-08-17T12:35:55.1071581Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ffa10aad08b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-08-17T12:35:55.1071977Z frame #13: + 0xdbbf4 (0x7ffa1093bbf4 in /opt/conda/lib/libstdc++.so.6) 2022-08-17T12:35:55.1072460Z frame #14: + 0x8609 (0x7ffa24e49609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-08-17T12:35:55.1072837Z frame #15: clone + 0x43 (0x7ffa24d6e133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-08-17T12:35:55.1073005Z 2022-08-17T12:35:55.1073308Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down 2022-08-17T12:35:55.3868388Z ok (5.084s) 2022-08-17T12:35:55.3868549Z 2022-08-17T12:35:55.3868921Z ---------------------------------------------------------------------- 2022-08-17T12:35:55.3869193Z Ran 1 test in 5.085s 2022-08-17T12:35:55.3869316Z 2022-08-17T12:35:55.3869367Z OK 2022-08-17T12:35:55.3869467Z 2022-08-17T12:35:55.3869563Z Generating XML reports... 2022-08-17T12:35:55.3908803Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123550.xml 2022-08-17T12:35:56.6933708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:56.6934112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:56.6936470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:56.6937058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:56.8469799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogjzs6gz 2022-08-17T12:35:56.8470606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogjzs6gz/_remote_module_non_scriptable.py 2022-08-17T12:35:57.1055961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:35:57.1066819Z 2022-08-17T12:35:57.1067047Z Running tests... 2022-08-17T12:35:57.1067646Z ---------------------------------------------------------------------- 2022-08-17T12:35:57.3702330Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32170 2022-08-17T12:35:57.3731572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32171 2022-08-17T12:35:57.3762824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32172 2022-08-17T12:35:57.3794868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32173 2022-08-17T12:35:58.4715546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:58.4716172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:58.4719220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:58.4720067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:58.4960250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:58.4960946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:58.4964271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:58.4964957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:58.4992692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:58.4993294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:58.4998243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:58.4998883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:58.5702452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:35:58.5703527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:35:58.5705816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:35:58.5706573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:35:58.6313455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0v9cavp 2022-08-17T12:35:58.6314145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0v9cavp/_remote_module_non_scriptable.py 2022-08-17T12:35:58.6568957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb81g3qb3 2022-08-17T12:35:58.6569825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb81g3qb3/_remote_module_non_scriptable.py 2022-08-17T12:35:58.6599583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm62ya3s 2022-08-17T12:35:58.6602284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm62ya3s/_remote_module_non_scriptable.py 2022-08-17T12:35:58.7287470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphluskhin 2022-08-17T12:35:58.7288909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphluskhin/_remote_module_non_scriptable.py 2022-08-17T12:35:58.8919616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:35:58.9192166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:35:58.9227410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:35:58.9914922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:05.3942278Z ok (8.287s) 2022-08-17T12:36:05.3942478Z 2022-08-17T12:36:05.3942891Z ---------------------------------------------------------------------- 2022-08-17T12:36:05.3943196Z Ran 1 test in 8.287s 2022-08-17T12:36:05.3943317Z 2022-08-17T12:36:05.3943379Z OK 2022-08-17T12:36:05.3943462Z 2022-08-17T12:36:05.3943555Z Generating XML reports... 2022-08-17T12:36:05.3978726Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123557.xml 2022-08-17T12:36:06.7206978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:06.7207690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:06.7210432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:06.7211024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:06.8749345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry2myjuv 2022-08-17T12:36:06.8750053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry2myjuv/_remote_module_non_scriptable.py 2022-08-17T12:36:07.1350750Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:07.1362407Z 2022-08-17T12:36:07.1362903Z Running tests... 2022-08-17T12:36:07.1363356Z ---------------------------------------------------------------------- 2022-08-17T12:36:07.4025058Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32345 2022-08-17T12:36:07.4053938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32346 2022-08-17T12:36:07.4084158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32347 2022-08-17T12:36:07.4118039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32348 2022-08-17T12:36:08.3993807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:08.3994782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:08.3997763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:08.3998439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:08.4671645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:08.4672296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:08.4674915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:08.4675499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:08.4714695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:08.4715352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:08.4719407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:08.4720022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:08.4957155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:08.4957795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:08.4961016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:08.4961679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:08.5576529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvjo6chm 2022-08-17T12:36:08.5577257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvjo6chm/_remote_module_non_scriptable.py 2022-08-17T12:36:08.6278281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuaohzinp 2022-08-17T12:36:08.6279127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuaohzinp/_remote_module_non_scriptable.py 2022-08-17T12:36:08.6311828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g6vkke0 2022-08-17T12:36:08.6313185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g6vkke0/_remote_module_non_scriptable.py 2022-08-17T12:36:08.6549106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ie19zau 2022-08-17T12:36:08.6549944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ie19zau/_remote_module_non_scriptable.py 2022-08-17T12:36:08.8191966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:08.8902876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:08.8921966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:08.9150894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:15.3266412Z ok (8.190s) 2022-08-17T12:36:15.3266584Z 2022-08-17T12:36:15.3267085Z ---------------------------------------------------------------------- 2022-08-17T12:36:15.3267354Z Ran 1 test in 8.190s 2022-08-17T12:36:15.3267463Z 2022-08-17T12:36:15.3267529Z OK 2022-08-17T12:36:15.3267625Z 2022-08-17T12:36:15.3267720Z Generating XML reports... 2022-08-17T12:36:15.3303825Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123607.xml 2022-08-17T12:36:16.6425470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:16.6428162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:16.6428908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:16.6429285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:16.7963024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphozbc_a8 2022-08-17T12:36:16.7964131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphozbc_a8/_remote_module_non_scriptable.py 2022-08-17T12:36:17.0542729Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:17.0554474Z 2022-08-17T12:36:17.0554594Z Running tests... 2022-08-17T12:36:17.0555647Z ---------------------------------------------------------------------- 2022-08-17T12:36:17.3197273Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32520 2022-08-17T12:36:17.3227155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32521 2022-08-17T12:36:17.3255972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32522 2022-08-17T12:36:17.3287172Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32523 2022-08-17T12:36:18.3229697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:18.3230331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:18.3232912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:18.3233489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:18.3385473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:18.3386129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:18.3388810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:18.3389444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:18.3763022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:18.3763564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:18.3766699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:18.3767246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:18.3822881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:18.3823429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:18.3827098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:18.3827844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:18.4830926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyfpqut1 2022-08-17T12:36:18.4832020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyfpqut1/_remote_module_non_scriptable.py 2022-08-17T12:36:18.4984069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_4r6pr_ 2022-08-17T12:36:18.4985327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_4r6pr_/_remote_module_non_scriptable.py 2022-08-17T12:36:18.5366484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0u5fw3r 2022-08-17T12:36:18.5368161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0u5fw3r/_remote_module_non_scriptable.py 2022-08-17T12:36:18.5452329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeog4y_bk 2022-08-17T12:36:18.5454017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeog4y_bk/_remote_module_non_scriptable.py 2022-08-17T12:36:18.7461783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:18.7621995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:18.8018041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:18.8133020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:25.2435478Z ok (8.188s) 2022-08-17T12:36:25.2435682Z 2022-08-17T12:36:25.2436153Z ---------------------------------------------------------------------- 2022-08-17T12:36:25.2436510Z Ran 1 test in 8.188s 2022-08-17T12:36:25.2436710Z 2022-08-17T12:36:25.2436810Z OK 2022-08-17T12:36:25.2436968Z 2022-08-17T12:36:25.2437092Z Generating XML reports... 2022-08-17T12:36:25.2475636Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123617.xml 2022-08-17T12:36:26.5330331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:26.5331069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:26.5333641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:26.5334139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:26.6847664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_wxuc0s 2022-08-17T12:36:26.6848864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_wxuc0s/_remote_module_non_scriptable.py 2022-08-17T12:36:26.9431649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:26.9442330Z 2022-08-17T12:36:26.9442484Z Running tests... 2022-08-17T12:36:26.9443112Z ---------------------------------------------------------------------- 2022-08-17T12:36:27.2082957Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32695 2022-08-17T12:36:27.2111514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32696 2022-08-17T12:36:27.2139657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32697 2022-08-17T12:36:27.2170084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32698 2022-08-17T12:36:28.2605671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:28.2606581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:28.2608941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:28.2609568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:28.2728731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:28.2729357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:28.2732661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:28.2733310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:28.3175161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:28.3175704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:28.3178827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:28.3179692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:28.3192018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:28.3192523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:28.3197005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:28.3198111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:28.4308034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z_rtjtj 2022-08-17T12:36:28.4309261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z_rtjtj/_remote_module_non_scriptable.py 2022-08-17T12:36:28.4330731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsttfqo99 2022-08-17T12:36:28.4332943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsttfqo99/_remote_module_non_scriptable.py 2022-08-17T12:36:28.4772159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_znt_yc 2022-08-17T12:36:28.4773281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_znt_yc/_remote_module_non_scriptable.py 2022-08-17T12:36:28.4969380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpojx_5s 2022-08-17T12:36:28.4970161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpojx_5s/_remote_module_non_scriptable.py 2022-08-17T12:36:28.6899129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:28.7073363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:28.7367730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:28.7528385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:35.2320454Z ok (8.287s) 2022-08-17T12:36:35.2320833Z 2022-08-17T12:36:35.2321242Z ---------------------------------------------------------------------- 2022-08-17T12:36:35.2321507Z Ran 1 test in 8.288s 2022-08-17T12:36:35.2321632Z 2022-08-17T12:36:35.2321681Z OK 2022-08-17T12:36:35.2321784Z 2022-08-17T12:36:35.2321891Z Generating XML reports... 2022-08-17T12:36:35.2360633Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123626.xml 2022-08-17T12:36:36.5257044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:36.5257451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:36.5259437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:36.5260015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:36.6797010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxh6b9gn 2022-08-17T12:36:36.6797644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxh6b9gn/_remote_module_non_scriptable.py 2022-08-17T12:36:36.9402175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:36.9412865Z 2022-08-17T12:36:36.9413275Z Running tests... 2022-08-17T12:36:36.9413688Z ---------------------------------------------------------------------- 2022-08-17T12:36:37.2090744Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 402 2022-08-17T12:36:37.2119993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 403 2022-08-17T12:36:37.2149283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 404 2022-08-17T12:36:37.2180602Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 405 2022-08-17T12:36:38.2041570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:38.2042263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:38.2045162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:38.2045809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:38.2360494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:38.2361175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:38.2363965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:38.2364596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:38.2369733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:38.2370373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:38.2374516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:38.2375199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:38.2665605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:38.2666150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:38.2669293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:38.2669804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:38.3634537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej0c3l97 2022-08-17T12:36:38.3635327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej0c3l97/_remote_module_non_scriptable.py 2022-08-17T12:36:38.3955191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped4sui3e 2022-08-17T12:36:38.3956500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped4sui3e/_remote_module_non_scriptable.py 2022-08-17T12:36:38.3967368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiopp6pxc 2022-08-17T12:36:38.3969509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiopp6pxc/_remote_module_non_scriptable.py 2022-08-17T12:36:38.4259271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbul3ob3o 2022-08-17T12:36:38.4260084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbul3ob3o/_remote_module_non_scriptable.py 2022-08-17T12:36:38.6251873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:38.6534342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:38.6553675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:38.6863809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:45.1325198Z ok (8.191s) 2022-08-17T12:36:45.1325420Z 2022-08-17T12:36:45.1325986Z ---------------------------------------------------------------------- 2022-08-17T12:36:45.1326320Z Ran 1 test in 8.191s 2022-08-17T12:36:45.1326444Z 2022-08-17T12:36:45.1326512Z OK 2022-08-17T12:36:45.1326610Z 2022-08-17T12:36:45.1326695Z Generating XML reports... 2022-08-17T12:36:45.1362492Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123636.xml 2022-08-17T12:36:46.4470552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:46.4471226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:46.4473640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:46.4474041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:46.6000820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcljkalru 2022-08-17T12:36:46.6001840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcljkalru/_remote_module_non_scriptable.py 2022-08-17T12:36:46.8599353Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:46.8610691Z 2022-08-17T12:36:46.8611193Z Running tests... 2022-08-17T12:36:46.8611624Z ---------------------------------------------------------------------- 2022-08-17T12:36:47.1246931Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 577 2022-08-17T12:36:47.1276179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 578 2022-08-17T12:36:47.1305329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 579 2022-08-17T12:36:47.1337257Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 580 2022-08-17T12:36:48.1965298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:48.1966131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:48.1969547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:48.1970254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:48.1992335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:48.1993039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:48.1997142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:48.1997831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:48.2791636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:48.2792358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:48.2795735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:48.2796384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:48.2919512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:48.2920250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:48.2922779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:48.2923391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:48.3563197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81i7yk3l 2022-08-17T12:36:48.3563943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81i7yk3l/_remote_module_non_scriptable.py 2022-08-17T12:36:48.3581732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs4aq8zm 2022-08-17T12:36:48.3583182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs4aq8zm/_remote_module_non_scriptable.py 2022-08-17T12:36:48.4371033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb8wnpbr 2022-08-17T12:36:48.4371868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb8wnpbr/_remote_module_non_scriptable.py 2022-08-17T12:36:48.4500123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7x_obq7 2022-08-17T12:36:48.4502108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7x_obq7/_remote_module_non_scriptable.py 2022-08-17T12:36:48.6164726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:48.6175094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:48.6972501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:48.7071765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:36:55.1484303Z ok (8.287s) 2022-08-17T12:36:55.1484710Z 2022-08-17T12:36:55.1485329Z ---------------------------------------------------------------------- 2022-08-17T12:36:55.1485782Z Ran 1 test in 8.287s 2022-08-17T12:36:55.1486209Z 2022-08-17T12:36:55.1486307Z OK 2022-08-17T12:36:55.1486462Z 2022-08-17T12:36:55.1486606Z Generating XML reports... 2022-08-17T12:36:55.1522931Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123646.xml 2022-08-17T12:36:56.4672289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:56.4672722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:56.4675420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:56.4675800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:56.6220437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwakoigjy 2022-08-17T12:36:56.6221638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwakoigjy/_remote_module_non_scriptable.py 2022-08-17T12:36:56.8837918Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:36:56.8849254Z 2022-08-17T12:36:56.8849361Z Running tests... 2022-08-17T12:36:56.8849860Z ---------------------------------------------------------------------- 2022-08-17T12:36:57.1534638Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 753 2022-08-17T12:36:57.1565291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 754 2022-08-17T12:36:57.1595163Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 755 2022-08-17T12:36:57.1628635Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 756 2022-08-17T12:36:58.2409154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:58.2409776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:58.2412325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:58.2412898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:58.2427463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:58.2428085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:58.2431591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:58.2432220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:58.2529070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:58.2529738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:58.2533143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:58.2533773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:58.3036022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:36:58.3036568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:36:58.3037818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:36:58.3038350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:36:58.4041882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpben2ml96 2022-08-17T12:36:58.4042710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpben2ml96/_remote_module_non_scriptable.py 2022-08-17T12:36:58.4148240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0kx2oe3 2022-08-17T12:36:58.4149631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0kx2oe3/_remote_module_non_scriptable.py 2022-08-17T12:36:58.4268461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrs_3f55 2022-08-17T12:36:58.4269759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrs_3f55/_remote_module_non_scriptable.py 2022-08-17T12:36:58.4845172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy7m94sd 2022-08-17T12:36:58.4846276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy7m94sd/_remote_module_non_scriptable.py 2022-08-17T12:36:58.6802627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:36:58.6893770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:36:58.6943945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:36:58.7489581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:01.2716425Z ok (4.386s) 2022-08-17T12:37:01.2716681Z 2022-08-17T12:37:01.2717220Z ---------------------------------------------------------------------- 2022-08-17T12:37:01.2717523Z Ran 1 test in 4.387s 2022-08-17T12:37:01.2717647Z 2022-08-17T12:37:01.2717714Z OK 2022-08-17T12:37:01.2717812Z 2022-08-17T12:37:01.2717893Z Generating XML reports... 2022-08-17T12:37:01.2756018Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123656.xml 2022-08-17T12:37:02.5664055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:02.5664452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:02.5667332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:02.5667713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:02.7199108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cwdqz_l 2022-08-17T12:37:02.7199972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cwdqz_l/_remote_module_non_scriptable.py 2022-08-17T12:37:02.9779178Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:37:02.9790482Z 2022-08-17T12:37:02.9790976Z Running tests... 2022-08-17T12:37:02.9791390Z ---------------------------------------------------------------------- 2022-08-17T12:37:03.2442654Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 928 2022-08-17T12:37:03.2470707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 929 2022-08-17T12:37:03.2498754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 930 2022-08-17T12:37:03.2529625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 931 2022-08-17T12:37:04.3022570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:04.3023253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:04.3025881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:04.3026534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:04.3512448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:04.3513092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:04.3516426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:04.3517099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:04.3684745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:04.3685694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:04.3688857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:04.3689507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:04.3768477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:04.3769176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:04.3772975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:04.3773740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:04.4609452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp4u1kf8 2022-08-17T12:37:04.4610354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp4u1kf8/_remote_module_non_scriptable.py 2022-08-17T12:37:04.5111758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsocok5uo 2022-08-17T12:37:04.5112540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsocok5uo/_remote_module_non_scriptable.py 2022-08-17T12:37:04.5294891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweth_dpm 2022-08-17T12:37:04.5295898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweth_dpm/_remote_module_non_scriptable.py 2022-08-17T12:37:04.5344442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ljx7ksm 2022-08-17T12:37:04.5345954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ljx7ksm/_remote_module_non_scriptable.py 2022-08-17T12:37:04.7239683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:04.7732179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:04.7934618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:04.7940073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:09.7654030Z ok (6.786s) 2022-08-17T12:37:09.7654307Z 2022-08-17T12:37:09.7654851Z ---------------------------------------------------------------------- 2022-08-17T12:37:09.7655143Z Ran 1 test in 6.786s 2022-08-17T12:37:09.7655267Z 2022-08-17T12:37:09.7655333Z OK 2022-08-17T12:37:09.7655430Z 2022-08-17T12:37:09.7655525Z Generating XML reports... 2022-08-17T12:37:09.7691370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123702.xml 2022-08-17T12:37:11.0760348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:11.0760730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:11.0763476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:11.0763857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:11.2304507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr12ba1qn 2022-08-17T12:37:11.2306074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr12ba1qn/_remote_module_non_scriptable.py 2022-08-17T12:37:11.4879812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-08-17T12:37:11.4890460Z 2022-08-17T12:37:11.4890556Z Running tests... 2022-08-17T12:37:11.4891417Z ---------------------------------------------------------------------- 2022-08-17T12:37:11.7503996Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1103 2022-08-17T12:37:11.7533398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1104 2022-08-17T12:37:11.7561793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1105 2022-08-17T12:37:11.7593205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1106 2022-08-17T12:37:12.7651627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:12.7654477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:12.7655362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:12.7655957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:12.7765793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:12.7766731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:12.7770274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:12.7770918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:12.7840560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:12.7844408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:12.7845283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:12.7846156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:12.7862679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:12.7863321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:12.7867725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:12.7868413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:12.9252879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_i925uf 2022-08-17T12:37:12.9254173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_i925uf/_remote_module_non_scriptable.py 2022-08-17T12:37:12.9367239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdl5ks3js 2022-08-17T12:37:12.9368040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdl5ks3js/_remote_module_non_scriptable.py 2022-08-17T12:37:12.9443453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhazy9gg 2022-08-17T12:37:12.9444634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhazy9gg/_remote_module_non_scriptable.py 2022-08-17T12:37:12.9471531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ean6lk2 2022-08-17T12:37:12.9474096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ean6lk2/_remote_module_non_scriptable.py 2022-08-17T12:37:13.1850280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:13.1940671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:13.2050118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:13.2083472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:18.2716275Z ok (6.782s) 2022-08-17T12:37:18.2716541Z 2022-08-17T12:37:18.2716935Z ---------------------------------------------------------------------- 2022-08-17T12:37:18.2717203Z Ran 1 test in 6.782s 2022-08-17T12:37:18.2717326Z 2022-08-17T12:37:18.2717377Z OK 2022-08-17T12:37:18.2718009Z 2022-08-17T12:37:18.2718155Z Generating XML reports... 2022-08-17T12:37:18.2754745Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123711.xml 2022-08-17T12:37:18.8586205Z Running distributed/rpc/test_share_memory ... [2022-08-17 12:37:18.858225] 2022-08-17T12:37:18.8586901Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:37:18.858309] 2022-08-17T12:37:19.9543597Z 2022-08-17T12:37:19.9543972Z ]> 2022-08-17T12:37:19.9544258Z test_case (__main__.TestRPCPickler) 2022-08-17T12:37:20.8892735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:20.8893446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:20.8896177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:20.8896777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:21.0405316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory 2022-08-17T12:37:21.0415930Z 2022-08-17T12:37:21.0416200Z Running tests... 2022-08-17T12:37:21.0416689Z ---------------------------------------------------------------------- 2022-08-17T12:37:22.2500209Z test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:22.2500652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:22.2502714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:22.2503141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:22.8430795Z Exception ignored in: 2022-08-17T12:37:22.8431236Z Traceback (most recent call last): 2022-08-17T12:37:22.8431937Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__ 2022-08-17T12:37:22.8432670Z File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref 2022-08-17T12:37:22.8433396Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref' 2022-08-17T12:37:22.8433717Z Exception ignored in: 2022-08-17T12:37:22.8433990Z Traceback (most recent call last): 2022-08-17T12:37:22.8434396Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__ 2022-08-17T12:37:22.8435067Z File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref 2022-08-17T12:37:22.8435456Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref' 2022-08-17T12:37:22.9647965Z ok (1.923s) 2022-08-17T12:37:22.9648341Z 2022-08-17T12:37:22.9649021Z ---------------------------------------------------------------------- 2022-08-17T12:37:22.9649463Z Ran 1 test in 1.923s 2022-08-17T12:37:22.9649642Z 2022-08-17T12:37:22.9649753Z OK 2022-08-17T12:37:22.9649913Z 2022-08-17T12:37:22.9650058Z Generating XML reports... 2022-08-17T12:37:22.9686510Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220817123721.xml 2022-08-17T12:37:23.5110074Z Running distributed/rpc/test_tensorpipe_agent ... [2022-08-17 12:37:23.510591] 2022-08-17T12:37:23.5110705Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 12:37:23.510687] 2022-08-17T12:37:24.6186420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjjffyg9 2022-08-17T12:37:24.6187591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjjffyg9/_remote_module_non_scriptable.py 2022-08-17T12:37:24.8809749Z , <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]> 2022-08-17T12:37:24.8810985Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) 2022-08-17T12:37:24.8811557Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) 2022-08-17T12:37:24.8812146Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) 2022-08-17T12:37:24.8812659Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) 2022-08-17T12:37:24.8813731Z , <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]> 2022-08-17T12:37:24.8814774Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-08-17T12:37:24.8815390Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-08-17T12:37:24.8816009Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-08-17T12:37:24.8816610Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) 2022-08-17T12:37:24.8826944Z , <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]> 2022-08-17T12:37:24.8837532Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8838111Z test_autograd_context (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8838651Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8839003Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8839533Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8839900Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8840357Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8840849Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8841363Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8841882Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8842397Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8842913Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8843466Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8843989Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8844566Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8845108Z test_backward_rref (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8845585Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8846155Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8846474Z test_backward_simple (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8846934Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8847435Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8847917Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8848307Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8848641Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8848951Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8849281Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8849603Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8849916Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8850254Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8850586Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8850914Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8851232Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8851575Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8851889Z test_debug_info (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8852186Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8852503Z test_error_in_context (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8852831Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8853169Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8853485Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8853813Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8854138Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8854454Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8854790Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8855250Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8855583Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8855919Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8856243Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8856569Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8856866Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8857190Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8857531Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8858020Z test_nested_context (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8858317Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8858623Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8858956Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8859306Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8859684Z test_post_hooks (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8859990Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8860284Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8860593Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8860899Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8861207Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8861534Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:37:24.8862204Z , <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]> 2022-08-17T12:37:24.8862852Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) 2022-08-17T12:37:24.8863151Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) 2022-08-17T12:37:24.8863496Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) 2022-08-17T12:37:24.8863836Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) 2022-08-17T12:37:24.8864527Z , <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]> 2022-08-17T12:37:24.8865189Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) 2022-08-17T12:37:24.8865491Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) 2022-08-17T12:37:24.8865812Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) 2022-08-17T12:37:24.8866171Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) 2022-08-17T12:37:24.8871534Z , <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]> 2022-08-17T12:37:24.8876594Z test_add_done_callback (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8876914Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8877254Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8877606Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8877909Z test_async_function_remote (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8878207Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8878494Z test_async_function_simple (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8878806Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8879132Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8879458Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8879754Z test_async_script_throw (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8880037Z test_async_script_udf (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8880336Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8880668Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8881006Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8881346Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8881719Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8882054Z test_callback_chain (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8882334Z test_callback_simple (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8882623Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8882919Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8883246Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8883566Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8883871Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8884185Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8884475Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8884779Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8885098Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8885406Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8885716Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8886192Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8886516Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8886857Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8887193Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8887488Z test_remote_script_module (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8887783Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8888069Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8888384Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8888753Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8889083Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8889379Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8889685Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8889962Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8890255Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8890542Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8890820Z test_rref_local_value (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8891175Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8891479Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8891793Z test_torchscript_function (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8892106Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8892426Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8892756Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8893061Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8893362Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) 2022-08-17T12:37:24.8893750Z ]> 2022-08-17T12:37:24.8894176Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) 2022-08-17T12:37:24.8894596Z ]> 2022-08-17T12:37:24.8894988Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) 2022-08-17T12:37:24.8896541Z , <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]> 2022-08-17T12:37:24.8898108Z test_bad_module (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8898412Z test_forward_async (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8898728Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8899040Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8899331Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8899642Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8899948Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8900267Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8900637Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8900983Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8901341Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8901695Z test_train_eval (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8902000Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) 2022-08-17T12:37:24.8920184Z , <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks_new_signatures>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]> 2022-08-17T12:37:24.8937528Z test_add (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8937802Z test_add_done_callback (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8938075Z test_add_with_id (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8938338Z test_all_gather (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8938590Z test_all_gather_timeout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8938856Z test_async_add (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8939118Z test_async_class_method (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8939390Z test_async_class_method_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8939673Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8939963Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8940244Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8940539Z test_async_function_chained (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8940831Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8941126Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8941421Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8941738Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8942046Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8942333Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8942643Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8942938Z test_async_function_nested (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8943214Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8943501Z test_async_function_raise (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8943850Z test_async_function_raise_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8944141Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8944414Z test_async_function_simple (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8944710Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8945025Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8945327Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8945645Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8945966Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8946272Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8946596Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8947022Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8947352Z test_async_static_method (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8947625Z test_async_static_method_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8947915Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8948245Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8948510Z test_builtin_remote_self (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8948784Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8949048Z test_callback_chain (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8949300Z test_callback_in_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8949563Z test_callback_multi (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8949825Z test_callback_none (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8950087Z test_callback_simple (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8950345Z test_callback_with_error (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8950623Z test_callback_with_ret (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8950901Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8951168Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8951470Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8951749Z test_deadlock (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8951990Z test_debug_info (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8952259Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8952540Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8952808Z test_dist_init_decorator (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8953080Z test_duplicate_name (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8953347Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8953615Z test_expected_src (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8953874Z test_function_not_on_callee (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8954143Z test_future_done (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8954417Z test_future_done_exception (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8954673Z test_future_in_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8954945Z test_future_nested_callback (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8955218Z test_future_wait_twice (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8955473Z test_get_worker_infos (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8955769Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8956076Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8956337Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8956606Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8956874Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8957176Z test_init_rpc_twice (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8957420Z test_int_callee (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8957678Z test_invalid_names (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8957947Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8958202Z test_local_shutdown (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8958478Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8958763Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8959022Z test_mark_future_twice (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8959299Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8959592Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8959860Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8960122Z test_multi_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8960390Z test_my_parameter_server (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8960668Z test_nested_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8960914Z test_nested_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8961170Z test_nested_rref (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8961437Z test_nested_rref_stress (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8961722Z test_non_cont_tensors (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8962055Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8962366Z test_nonzero (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8962607Z test_owner_equality (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8962876Z test_owner_rref_backward (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8963146Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8963405Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8963674Z test_pickle_future (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8963949Z test_profiler_export_trace (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8964244Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8964558Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8964915Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8965198Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8965472Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8965769Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8966259Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8966585Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8966889Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8967209Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8967541Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8967852Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8968171Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8968489Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8968781Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8969100Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8969422Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8969743Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8970053Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8970375Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8970754Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8971062Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8971380Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8971700Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8972015Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8972295Z test_py_built_in (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8972565Z test_py_class_constructor (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8972852Z test_py_class_instance_method (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8973114Z test_py_class_method (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8973388Z test_py_class_static_method (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8973668Z test_py_function_exception (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8973931Z test_py_multi_async_call (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8974203Z test_py_nested_pickle (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8974476Z test_py_no_return_result (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8974737Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8975064Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8975350Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8975613Z test_py_rref_args (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8975870Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8976136Z test_py_tensors (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8976409Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8976684Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8976955Z test_py_udf_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8977219Z test_py_user_defined (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8977520Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8977819Z test_reinit (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8978083Z test_remote_same_worker (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8978348Z test_remote_throw (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8978606Z test_remote_with_exception (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8978875Z test_return_future (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8979141Z test_return_future_async (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8979402Z test_return_future_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8979674Z test_return_local_rrefs (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8979939Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8980210Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8980504Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8980787Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8981060Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8981382Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8981714Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8982002Z test_rpc_return_rref (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8982251Z test_rpc_timeouts (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8982523Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8982796Z test_rref_forward_chain (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8983055Z test_rref_get_future (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8983311Z test_rref_leak (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8983581Z test_rref_proxy_class (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8983840Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8984149Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8984418Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8984671Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8984943Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8985236Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8985493Z test_rref_str (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8985741Z test_rref_timeout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8986003Z test_rref_type_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8986278Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8986546Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8986912Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8987195Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8987472Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8987774Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8988059Z test_scalar_add (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8988297Z test_self_add (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8988609Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8988894Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8989173Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8989471Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8989772Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8990049Z test_send_to_rank (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8990318Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8990620Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8990906Z test_shutdown_errors (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8991170Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8991449Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8991738Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8992005Z test_stress_light_rpc (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8992272Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8992546Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8992825Z test_user_rref_backward (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8993087Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8993371Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8993634Z test_wait_all (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8993891Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8994182Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8994484Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8994767Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8995043Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8995323Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8995596Z test_wait_all_timeout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8995856Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8996146Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8996434Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8996704Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8996992Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8997261Z test_worker_id (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8997544Z test_worker_info_pickle (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8997808Z test_world_size_one (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.8998064Z test_wrong_types (__main__.TensorPipeRpcTest) 2022-08-17T12:37:24.9002845Z , <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]> 2022-08-17T12:37:24.9016986Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9017441Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9017875Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9018280Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9018692Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9019242Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9019656Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9020077Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9020498Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9020906Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9021308Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9021737Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9022165Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9022600Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9023022Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9023441Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9023901Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9024324Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9024744Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9025179Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9025617Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9026025Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9026442Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9026965Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9027381Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9027797Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9028251Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9028705Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9029119Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9029526Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9029926Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) 2022-08-17T12:37:24.9034917Z , <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_and_static_init_rpc_together>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc_without_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]> 2022-08-17T12:37:24.9039782Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9040149Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9040523Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9040921Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9041358Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9041749Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9042116Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9042508Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9042899Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9043254Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9043622Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9043986Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9044332Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9044714Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9045067Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9045411Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9045740Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9046253Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9046609Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9046951Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9047303Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9047660Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9048034Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9048399Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9048771Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9049179Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9049518Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9049891Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9050272Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9050656Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9051029Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9051392Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9051755Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9052102Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9052490Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9052880Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9053241Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9053589Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9053944Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) 2022-08-17T12:37:24.9054678Z , <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]> 2022-08-17T12:37:24.9055422Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-08-17T12:37:24.9055819Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-08-17T12:37:24.9056251Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) 2022-08-17T12:37:25.8182184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:25.8182588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:25.8187174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:25.8187552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:25.9749775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6glbwa4_ 2022-08-17T12:37:25.9752626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6glbwa4_/_remote_module_non_scriptable.py 2022-08-17T12:37:26.2370704Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:26.2381416Z 2022-08-17T12:37:26.2381668Z Running tests... 2022-08-17T12:37:26.2382297Z ---------------------------------------------------------------------- 2022-08-17T12:37:26.4882387Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.250s) 2022-08-17T12:37:26.4882900Z 2022-08-17T12:37:26.4883133Z ---------------------------------------------------------------------- 2022-08-17T12:37:26.4883390Z Ran 1 test in 0.250s 2022-08-17T12:37:26.4883512Z 2022-08-17T12:37:26.4883588Z OK (skipped=1) 2022-08-17T12:37:26.4883702Z 2022-08-17T12:37:26.4883790Z Generating XML reports... 2022-08-17T12:37:26.4910156Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123726.xml 2022-08-17T12:37:27.7146971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:27.7147415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:27.7150167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:27.7150585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:27.8682658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lo61paz 2022-08-17T12:37:27.8685520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lo61paz/_remote_module_non_scriptable.py 2022-08-17T12:37:28.1363652Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:28.1375083Z 2022-08-17T12:37:28.1375185Z Running tests... 2022-08-17T12:37:28.1375634Z ---------------------------------------------------------------------- 2022-08-17T12:37:28.3940645Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s) 2022-08-17T12:37:28.3941278Z 2022-08-17T12:37:28.3941501Z ---------------------------------------------------------------------- 2022-08-17T12:37:28.3941759Z Ran 1 test in 0.256s 2022-08-17T12:37:28.3941895Z 2022-08-17T12:37:28.3941974Z OK (skipped=1) 2022-08-17T12:37:28.3942088Z 2022-08-17T12:37:28.3942178Z Generating XML reports... 2022-08-17T12:37:28.3966883Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123728.xml 2022-08-17T12:37:29.6454341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:29.6454776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:29.6456880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:29.6457353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:29.7949314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4slw2_lq 2022-08-17T12:37:29.7952527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4slw2_lq/_remote_module_non_scriptable.py 2022-08-17T12:37:30.0593688Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:30.0604194Z 2022-08-17T12:37:30.0604617Z Running tests... 2022-08-17T12:37:30.0605021Z ---------------------------------------------------------------------- 2022-08-17T12:37:30.3219170Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1429 2022-08-17T12:37:30.3247210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1430 2022-08-17T12:37:30.3276727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1431 2022-08-17T12:37:30.3308811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1432 2022-08-17T12:37:31.3490311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:31.3493745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:31.3494630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:31.3495264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:31.3827177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:31.3827872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:31.3830154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:31.3830816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:31.3831661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:31.3832266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:31.3833081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:31.3833696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:31.3836786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:31.3837410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:31.3838169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:31.3838777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:31.5081600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub_0b457 2022-08-17T12:37:31.5082298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub_0b457/_remote_module_non_scriptable.py 2022-08-17T12:37:31.5416353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxn_54xx 2022-08-17T12:37:31.5417143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxn_54xx/_remote_module_non_scriptable.py 2022-08-17T12:37:31.5430685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xr8xkls 2022-08-17T12:37:31.5433521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xr8xkls/_remote_module_non_scriptable.py 2022-08-17T12:37:31.5443973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7157hg3f 2022-08-17T12:37:31.5445673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7157hg3f/_remote_module_non_scriptable.py 2022-08-17T12:37:31.7710355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:31.8032759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:31.8040484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:31.8061630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:32.0648256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:32.0749364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:32.0855363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:32.0856684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:32.0858041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:32.0859265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:32.0861204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:32.0863442Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:32.1253441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1254563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1255755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1256676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1400285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1401478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1402785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.1404039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:32.5368887Z ok (2.476s) 2022-08-17T12:37:32.5369139Z 2022-08-17T12:37:32.5369699Z ---------------------------------------------------------------------- 2022-08-17T12:37:32.5369997Z Ran 1 test in 2.476s 2022-08-17T12:37:32.5370117Z 2022-08-17T12:37:32.5370182Z OK 2022-08-17T12:37:32.5370278Z 2022-08-17T12:37:32.5370374Z Generating XML reports... 2022-08-17T12:37:32.5403426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123730.xml 2022-08-17T12:37:33.8439803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:33.8440207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:33.8442728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:33.8443104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:33.9986044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc9ysbq5 2022-08-17T12:37:33.9988648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc9ysbq5/_remote_module_non_scriptable.py 2022-08-17T12:37:34.2686931Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:34.2698283Z 2022-08-17T12:37:34.2698424Z Running tests... 2022-08-17T12:37:34.2699049Z ---------------------------------------------------------------------- 2022-08-17T12:37:34.5382704Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1684 2022-08-17T12:37:34.5413259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1685 2022-08-17T12:37:34.5442754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1686 2022-08-17T12:37:34.5473443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1687 2022-08-17T12:37:35.5976552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:35.5977257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:35.5980334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:35.5981023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:35.6079607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:35.6080251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:35.6083711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:35.6084383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:35.6315954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:35.6316666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:35.6319022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:35.6319688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:35.6372258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:35.6372976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:35.6376044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:35.6376764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:35.7580361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch9a58gc 2022-08-17T12:37:35.7581151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch9a58gc/_remote_module_non_scriptable.py 2022-08-17T12:37:35.7701202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54hpfk74 2022-08-17T12:37:35.7702745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54hpfk74/_remote_module_non_scriptable.py 2022-08-17T12:37:35.7913124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjnnu3q_ 2022-08-17T12:37:35.7914729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjnnu3q_/_remote_module_non_scriptable.py 2022-08-17T12:37:35.7970367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm6y_7tn 2022-08-17T12:37:35.7971756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm6y_7tn/_remote_module_non_scriptable.py 2022-08-17T12:37:36.0207786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:36.0355642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:36.0526739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:36.0591763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:36.3715672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:36.3716620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:36.3815534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:36.3816827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:36.3818299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:36.3820379Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:36.3823807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:36.3825287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:37:36.4796704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:36.4893281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:36.4971895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:36.5051840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:36.9536868Z ok (2.684s) 2022-08-17T12:37:36.9537148Z 2022-08-17T12:37:36.9537638Z ---------------------------------------------------------------------- 2022-08-17T12:37:36.9538357Z Ran 1 test in 2.684s 2022-08-17T12:37:36.9538544Z 2022-08-17T12:37:36.9538643Z OK 2022-08-17T12:37:36.9538776Z 2022-08-17T12:37:36.9538925Z Generating XML reports... 2022-08-17T12:37:36.9572922Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123734.xml 2022-08-17T12:37:38.2602693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:38.2603078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:38.2605077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:38.2605474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:38.4099454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98dz8rju 2022-08-17T12:37:38.4102695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98dz8rju/_remote_module_non_scriptable.py 2022-08-17T12:37:38.6737690Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:38.6748470Z 2022-08-17T12:37:38.6748616Z Running tests... 2022-08-17T12:37:38.6749241Z ---------------------------------------------------------------------- 2022-08-17T12:37:38.9414155Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1951 2022-08-17T12:37:38.9442738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1952 2022-08-17T12:37:38.9473036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1953 2022-08-17T12:37:38.9503852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1954 2022-08-17T12:37:38.9535551Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1955 2022-08-17T12:37:38.9572517Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1956 2022-08-17T12:37:40.2102363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.2103048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.2106997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.2107818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.2213206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.2214168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.2218385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.2219358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.3621695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.3626614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.3629728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.3632624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.3908238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.3908810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.3912438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.3913050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.3945588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4v6b9y5 2022-08-17T12:37:40.3948206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4v6b9y5/_remote_module_non_scriptable.py 2022-08-17T12:37:40.4120735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7550eihh 2022-08-17T12:37:40.4123249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7550eihh/_remote_module_non_scriptable.py 2022-08-17T12:37:40.5239369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.5242435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.5243392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.5244046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.5482209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:40.5482870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:40.5486340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:40.5486992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:40.6080355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wszcobh 2022-08-17T12:37:40.6081449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wszcobh/_remote_module_non_scriptable.py 2022-08-17T12:37:40.6348505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gsdotnx 2022-08-17T12:37:40.6350323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gsdotnx/_remote_module_non_scriptable.py 2022-08-17T12:37:40.6833632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp7_aeci 2022-08-17T12:37:40.6835627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp7_aeci/_remote_module_non_scriptable.py 2022-08-17T12:37:40.7062471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_g0zbv_ 2022-08-17T12:37:40.7063933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_g0zbv_/_remote_module_non_scriptable.py 2022-08-17T12:37:40.8227084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-08-17T12:37:40.8346490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:40.9483788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-08-17T12:37:40.9709213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:40.9714324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:40.9862492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:41.2814255Z 2022-08-17 12:37:41,280 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-08-17T12:37:41.2815557Z 2022-08-17 12:37:41,280 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-08-17T12:37:41.3006157Z 2022-08-17 12:37:41,299 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-08-17T12:37:41.3012113Z 2022-08-17 12:37:41,300 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-08-17T12:37:41.3105692Z 2022-08-17 12:37:41,310 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-08-17T12:37:41.3111881Z 2022-08-17 12:37:41,310 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-08-17T12:37:41.3205399Z 2022-08-17 12:37:41,319 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-08-17T12:37:41.3212880Z 2022-08-17 12:37:41,320 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-08-17T12:37:41.3216261Z 2022-08-17 12:37:41,321 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-08-17T12:37:41.3230498Z 2022-08-17 12:37:41,322 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-08-17T12:37:41.3437493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:41.3438557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-08-17T12:37:41.3535119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:41.3637225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:41.3740384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-08-17T12:37:41.3750436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:41.3758072Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3765771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3775181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3783121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3845672Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3851292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:41.3852556Z 2022-08-17 12:37:41,384 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-08-17T12:37:41.3878247Z 2022-08-17 12:37:41,387 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-08-17T12:37:41.3882112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-08-17T12:37:41.3887178Z 2022-08-17 12:37:41,386 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-08-17T12:37:41.3894841Z 2022-08-17 12:37:41,388 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-08-17T12:37:41.3897517Z 2022-08-17 12:37:41,388 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-08-17T12:37:41.3912329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-08-17T12:37:41.4130974Z 2022-08-17 12:37:41,412 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-08-17T12:37:41.4135207Z 2022-08-17 12:37:41,412 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-08-17T12:37:41.5699239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T12:37:41.5798996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T12:37:41.5902160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-08-17T12:37:41.5906392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.5910030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T12:37:41.5913855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.5917884Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.5919267Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.5923172Z 2022-08-17 12:37:41,591 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net. 2022-08-17T12:37:41.5927385Z 2022-08-17 12:37:41,591 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net. 2022-08-17T12:37:41.5963801Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.5988616Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:41.6017686Z 2022-08-17 12:37:41,601 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net. 2022-08-17T12:37:41.6018721Z 2022-08-17 12:37:41,601 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net. 2022-08-17T12:37:41.6260078Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:41.6261500Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:41.6263245Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:41.6264583Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:41.6292185Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:41.6293608Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:41.6312034Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:41.6313630Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:41.6334498Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:41.6335574Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:41.6336692Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:41.6338015Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-08-17T12:37:41.6339404Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-08-17T12:37:41.6341080Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:41.6343022Z 2022-08-17 12:37:41,633 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-08-17T12:37:41.6365029Z 2022-08-17 12:37:41,636 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params. 2022-08-17T12:37:41.7004228Z 2022-08-17 12:37:41,699 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:41.7004833Z [ 1., -1.], 2022-08-17T12:37:41.7005670Z [ 1., 1.], 2022-08-17T12:37:41.7007087Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7008087Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7008757Z tensor([[-1., 1.], 2022-08-17T12:37:41.7009327Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:41.7009725Z [ 0., -16.]])} 2022-08-17T12:37:41.7010821Z 2022-08-17 12:37:41,699 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:41.7011584Z [-1., -1.], 2022-08-17T12:37:41.7011998Z [-1., 1.], 2022-08-17T12:37:41.7012927Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7013909Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7014513Z tensor([[-1., 1.], 2022-08-17T12:37:41.7015139Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:41.7015670Z [ 16., -16.]])} 2022-08-17T12:37:41.7016668Z 2022-08-17 12:37:41,699 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:41.7017348Z [ 1., 1.], 2022-08-17T12:37:41.7017788Z [ 1., -1.], 2022-08-17T12:37:41.7018686Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7019637Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7020246Z tensor([[-1., 1.], 2022-08-17T12:37:41.7020726Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:41.7021158Z [ 0., 16.]])} 2022-08-17T12:37:41.7056274Z 2022-08-17 12:37:41,705 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:41.7057056Z [-1., 1.], 2022-08-17T12:37:41.7058888Z [-1., -1.], 2022-08-17T12:37:41.7062694Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7066600Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7070063Z tensor([[-1., 1.], 2022-08-17T12:37:41.7073481Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:41.7076885Z [-16., 16.]])} 2022-08-17T12:37:41.7240876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:41.7276258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:41.7280012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:41.7287008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:41.7329426Z 2022-08-17 12:37:41,732 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:41.7333053Z [ 1., -1.], 2022-08-17T12:37:41.7336279Z [ 1., 1.], 2022-08-17T12:37:41.7339453Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7340654Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7341663Z tensor([[-1., 1.], 2022-08-17T12:37:41.7342590Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:41.7343584Z [ 0., -16.]])} 2022-08-17T12:37:41.7344521Z 2022-08-17 12:37:41,732 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:41.7345305Z [-1., 1.], 2022-08-17T12:37:41.7345767Z [-1., -1.], 2022-08-17T12:37:41.7347072Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7348066Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7348687Z tensor([[-1., 1.], 2022-08-17T12:37:41.7349304Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:41.7349821Z [-16., 16.]])} 2022-08-17T12:37:41.7360346Z 2022-08-17 12:37:41,735 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:41.7361137Z [-1., -1.], 2022-08-17T12:37:41.7361558Z [-1., 1.], 2022-08-17T12:37:41.7362516Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7363508Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7364126Z tensor([[-1., 1.], 2022-08-17T12:37:41.7364740Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:41.7365268Z [ 16., -16.]])} 2022-08-17T12:37:41.7373383Z 2022-08-17 12:37:41,736 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:41.7373972Z [ 1., 1.], 2022-08-17T12:37:41.7374318Z [ 1., -1.], 2022-08-17T12:37:41.7375049Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7376015Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7376908Z tensor([[-1., 1.], 2022-08-17T12:37:41.7377391Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:41.7377860Z [ 0., 16.]])} 2022-08-17T12:37:41.7585882Z 2022-08-17 12:37:41,757 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:41.7586671Z [ 1., -1.], 2022-08-17T12:37:41.7587181Z [ 1., 1.], 2022-08-17T12:37:41.7588125Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7589115Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7589723Z tensor([[-1., 1.], 2022-08-17T12:37:41.7590353Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:41.7590894Z [ 0., -16.]])} 2022-08-17T12:37:41.7591915Z 2022-08-17 12:37:41,757 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:41.7592660Z [-1., 1.], 2022-08-17T12:37:41.7593099Z [-1., -1.], 2022-08-17T12:37:41.7594326Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7595297Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7595906Z tensor([[-1., 1.], 2022-08-17T12:37:41.7596527Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:41.7597047Z [-16., 16.]])} 2022-08-17T12:37:41.7597975Z 2022-08-17 12:37:41,758 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:41.7598650Z [ 1., 1.], 2022-08-17T12:37:41.7599069Z [ 1., -1.], 2022-08-17T12:37:41.7599961Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7600928Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7601519Z tensor([[-1., 1.], 2022-08-17T12:37:41.7601999Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:41.7602450Z [ 0., 16.]])} 2022-08-17T12:37:41.7603791Z 2022-08-17 12:37:41,759 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:41.7604541Z [-1., -1.], 2022-08-17T12:37:41.7604987Z [-1., 1.], 2022-08-17T12:37:41.7606037Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:41.7607036Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:41.7607635Z tensor([[-1., 1.], 2022-08-17T12:37:41.7608250Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:41.7608786Z [ 16., -16.]])} 2022-08-17T12:37:41.7674982Z 2022-08-17 12:37:41,766 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-08-17T12:37:41.7681663Z 2022-08-17 12:37:41,767 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-08-17T12:37:41.7689557Z 2022-08-17 12:37:41,768 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-08-17T12:37:41.7697357Z 2022-08-17 12:37:41,769 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-08-17T12:37:41.7705402Z 2022-08-17 12:37:41,770 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-08-17T12:37:42.3659261Z ok (3.691s) 2022-08-17T12:37:42.3659520Z 2022-08-17T12:37:42.3660083Z ---------------------------------------------------------------------- 2022-08-17T12:37:42.3660423Z Ran 1 test in 3.691s 2022-08-17T12:37:42.3660548Z 2022-08-17T12:37:42.3660611Z OK 2022-08-17T12:37:42.3660711Z 2022-08-17T12:37:42.3660825Z Generating XML reports... 2022-08-17T12:37:42.3694157Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123738.xml 2022-08-17T12:37:43.6737389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:43.6737783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:43.6740553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:43.6740933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:43.8281693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qksqdch 2022-08-17T12:37:43.8285326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qksqdch/_remote_module_non_scriptable.py 2022-08-17T12:37:44.0951326Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:44.0961520Z 2022-08-17T12:37:44.0962093Z Running tests... 2022-08-17T12:37:44.0962691Z ---------------------------------------------------------------------- 2022-08-17T12:37:44.3598259Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2365 2022-08-17T12:37:44.3627625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2366 2022-08-17T12:37:44.3655899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2367 2022-08-17T12:37:44.3686907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2368 2022-08-17T12:37:44.3718196Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2369 2022-08-17T12:37:44.3754272Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2370 2022-08-17T12:37:45.5944324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.5944972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.5948542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.5949496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:45.6713280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.6714116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.6715049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.6715802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:45.8210692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_lfsljq 2022-08-17T12:37:45.8214879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_lfsljq/_remote_module_non_scriptable.py 2022-08-17T12:37:45.8338575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3d5fbz9 2022-08-17T12:37:45.8339503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3d5fbz9/_remote_module_non_scriptable.py 2022-08-17T12:37:45.8908441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.8909154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.8912185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.8913191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:45.9344794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.9345511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.9348885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.9349621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:45.9560230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.9560816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.9564808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.9565388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:45.9847811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:45.9848779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:45.9850339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:45.9850969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:46.0512522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6cxxsna 2022-08-17T12:37:46.0514608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6cxxsna/_remote_module_non_scriptable.py 2022-08-17T12:37:46.1055321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:46.1823437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7md69k6 2022-08-17T12:37:46.1824382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7md69k6/_remote_module_non_scriptable.py 2022-08-17T12:37:46.2037637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryrqrqu5 2022-08-17T12:37:46.2039621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryrqrqu5/_remote_module_non_scriptable.py 2022-08-17T12:37:46.2197580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nn41_ty 2022-08-17T12:37:46.2198731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nn41_ty/_remote_module_non_scriptable.py 2022-08-17T12:37:46.2389707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:46.3219630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:46.4878824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-08-17T12:37:46.5175704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:46.5314281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-08-17T12:37:46.8600482Z 2022-08-17 12:37:46,859 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-08-17T12:37:46.8602745Z 2022-08-17 12:37:46,859 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-08-17T12:37:46.8613141Z 2022-08-17 12:37:46,860 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-08-17T12:37:46.8615870Z 2022-08-17 12:37:46,860 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-08-17T12:37:46.8701915Z 2022-08-17 12:37:46,869 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-08-17T12:37:46.8703564Z 2022-08-17 12:37:46,869 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-08-17T12:37:46.8803574Z 2022-08-17 12:37:46,879 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-08-17T12:37:46.8804819Z 2022-08-17 12:37:46,879 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-08-17T12:37:46.8902777Z 2022-08-17 12:37:46,889 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-08-17T12:37:46.8904228Z 2022-08-17 12:37:46,889 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-08-17T12:37:46.9223328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:46.9326960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:46.9428961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:46.9530039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:46.9635968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-08-17T12:37:46.9636876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-08-17T12:37:46.9638210Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9643182Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9645436Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9646812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9659049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9660322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:46.9661532Z 2022-08-17 12:37:46,965 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-08-17T12:37:46.9738413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-08-17T12:37:46.9752338Z 2022-08-17 12:37:46,974 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-08-17T12:37:46.9761796Z 2022-08-17 12:37:46,975 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-08-17T12:37:46.9762981Z 2022-08-17 12:37:46,974 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-08-17T12:37:46.9769990Z 2022-08-17 12:37:46,976 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-08-17T12:37:46.9784181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-08-17T12:37:47.0096121Z 2022-08-17 12:37:47,008 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-08-17T12:37:47.0100503Z 2022-08-17 12:37:47,009 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-08-17T12:37:47.1535843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T12:37:47.1641345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T12:37:47.1750737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-08-17T12:37:47.1751656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T12:37:47.1753017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1754242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1760208Z 2022-08-17 12:37:47,175 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:47.1761756Z 2022-08-17 12:37:47,175 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:47.1773863Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1785931Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1799800Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1809805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:47.1843840Z 2022-08-17 12:37:47,183 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:47.1845032Z 2022-08-17 12:37:47,183 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:47.1854384Z 2022-08-17 12:37:47,184 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:47.1860464Z 2022-08-17 12:37:47,185 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:47.1882513Z 2022-08-17 12:37:47,187 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:47.1883702Z 2022-08-17 12:37:47,187 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:47.1988083Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:47.1988918Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:47.2112934Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:47.2113787Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:47.2137657Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:47.2138455Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:47.2139628Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:47.2140322Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:47.2164995Z 2022-08-17 12:37:47,216 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:47.2174843Z 2022-08-17 12:37:47,217 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:47.2207555Z 2022-08-17 12:37:47,220 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:47.2208388Z 2022-08-17 12:37:47,220 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:47.2707636Z 2022-08-17 12:37:47,269 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:47.2708553Z [-1., -1.], 2022-08-17T12:37:47.2709063Z [-1., 1.], 2022-08-17T12:37:47.2710012Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.2711036Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.2711623Z tensor([[-1., 1.], 2022-08-17T12:37:47.2712106Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.2712562Z [0., 0.]])} 2022-08-17T12:37:47.2713946Z 2022-08-17 12:37:47,269 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:47.2714631Z [ 1., 1.], 2022-08-17T12:37:47.2715054Z [ 1., -1.], 2022-08-17T12:37:47.2715958Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.2716922Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.2717531Z tensor([[-1., 1.], 2022-08-17T12:37:47.2717985Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.2718447Z [0., 0.]])} 2022-08-17T12:37:47.2719445Z 2022-08-17 12:37:47,269 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:47.2720168Z [ 1., -1.], 2022-08-17T12:37:47.2720532Z [ 1., 1.], 2022-08-17T12:37:47.2721456Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.2722420Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.2723032Z tensor([[-1., 1.], 2022-08-17T12:37:47.2723507Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.2723959Z [0., 0.]])} 2022-08-17T12:37:47.2748104Z 2022-08-17 12:37:47,274 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:47.2748883Z [-1., 1.], 2022-08-17T12:37:47.2750288Z [-1., -1.], 2022-08-17T12:37:47.2751480Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.2752479Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.2753080Z tensor([[-1., 1.], 2022-08-17T12:37:47.2754978Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.2755439Z [0., 0.]])} 2022-08-17T12:37:47.2800737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:47.2809642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:47.2818081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:47.2820411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:47.3029774Z 2022-08-17 12:37:47,302 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:47.3030330Z [-1., -1.], 2022-08-17T12:37:47.3030651Z [-1., 1.], 2022-08-17T12:37:47.3031422Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3032166Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3032561Z tensor([[-1., 1.], 2022-08-17T12:37:47.3032874Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3033178Z [0., 0.]])} 2022-08-17T12:37:47.3033829Z 2022-08-17 12:37:47,302 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:47.3034280Z [ 1., 1.], 2022-08-17T12:37:47.3034603Z [ 1., -1.], 2022-08-17T12:37:47.3035211Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3035884Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3036649Z tensor([[-1., 1.], 2022-08-17T12:37:47.3037003Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3037311Z [0., 0.]])} 2022-08-17T12:37:47.3038017Z 2022-08-17 12:37:47,302 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:47.3038544Z [ 1., -1.], 2022-08-17T12:37:47.3038775Z [ 1., 1.], 2022-08-17T12:37:47.3039450Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3040184Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3040619Z tensor([[-1., 1.], 2022-08-17T12:37:47.3040975Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3041325Z [0., 0.]])} 2022-08-17T12:37:47.3042128Z 2022-08-17 12:37:47,302 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:47.3042676Z [-1., 1.], 2022-08-17T12:37:47.3042995Z [-1., -1.], 2022-08-17T12:37:47.3043614Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3044291Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3044776Z tensor([[-1., 1.], 2022-08-17T12:37:47.3045137Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3045461Z [0., 0.]])} 2022-08-17T12:37:47.3252625Z 2022-08-17 12:37:47,324 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:47.3253456Z [-1., -1.], 2022-08-17T12:37:47.3253913Z [-1., 1.], 2022-08-17T12:37:47.3254897Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3255735Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3256164Z tensor([[-1., 1.], 2022-08-17T12:37:47.3256521Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3256805Z [0., 0.]])} 2022-08-17T12:37:47.3257495Z 2022-08-17 12:37:47,324 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:47.3258342Z [ 1., -1.], 2022-08-17T12:37:47.3277127Z [ 1., 1.], 2022-08-17T12:37:47.3278391Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3279398Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3280019Z tensor([[-1., 1.], 2022-08-17T12:37:47.3280489Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3280952Z [0., 0.]])} 2022-08-17T12:37:47.3282029Z 2022-08-17 12:37:47,326 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:47.3282770Z [-1., 1.], 2022-08-17T12:37:47.3283193Z [-1., -1.], 2022-08-17T12:37:47.3284109Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3285100Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3285694Z tensor([[-1., 1.], 2022-08-17T12:37:47.3286510Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3286972Z [0., 0.]])} 2022-08-17T12:37:47.3292885Z 2022-08-17 12:37:47,328 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:47.3293575Z [ 1., 1.], 2022-08-17T12:37:47.3293997Z [ 1., -1.], 2022-08-17T12:37:47.3294940Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:47.3295892Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing: 2022-08-17T12:37:47.3296512Z tensor([[-1., 1.], 2022-08-17T12:37:47.3296997Z [ 1., 1.]], requires_grad=True): tensor([[0., 0.], 2022-08-17T12:37:47.3297458Z [0., 0.]])} 2022-08-17T12:37:47.3367407Z 2022-08-17 12:37:47,336 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-08-17T12:37:47.3376461Z 2022-08-17 12:37:47,337 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-08-17T12:37:47.3384591Z 2022-08-17 12:37:47,338 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-08-17T12:37:47.3393240Z 2022-08-17 12:37:47,338 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-08-17T12:37:47.3403753Z 2022-08-17 12:37:47,339 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-08-17T12:37:47.8846620Z ok (3.788s) 2022-08-17T12:37:47.8846879Z 2022-08-17T12:37:47.8847268Z ---------------------------------------------------------------------- 2022-08-17T12:37:47.8847552Z Ran 1 test in 3.788s 2022-08-17T12:37:47.8847674Z 2022-08-17T12:37:47.8847739Z OK 2022-08-17T12:37:47.8847835Z 2022-08-17T12:37:47.8847934Z Generating XML reports... 2022-08-17T12:37:47.8880874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123744.xml 2022-08-17T12:37:49.2240357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:49.2240851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:49.2243642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:49.2244046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:49.3775502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiwqxj8w 2022-08-17T12:37:49.3778588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiwqxj8w/_remote_module_non_scriptable.py 2022-08-17T12:37:49.6419598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:49.6430881Z 2022-08-17T12:37:49.6431144Z Running tests... 2022-08-17T12:37:49.6431581Z ---------------------------------------------------------------------- 2022-08-17T12:37:49.9097552Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2779 2022-08-17T12:37:49.9125668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2780 2022-08-17T12:37:49.9154724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2781 2022-08-17T12:37:49.9191735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2782 2022-08-17T12:37:49.9230445Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2783 2022-08-17T12:37:49.9267240Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2784 2022-08-17T12:37:51.1123230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.1123851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.1124980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.1125652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.2646145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.2646842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.2647690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.2648341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.2721272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3d85t5c 2022-08-17T12:37:51.2724388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3d85t5c/_remote_module_non_scriptable.py 2022-08-17T12:37:51.4176252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.4176946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.4177806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.4178484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.4244565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyavpvss 2022-08-17T12:37:51.4246235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyavpvss/_remote_module_non_scriptable.py 2022-08-17T12:37:51.5387169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.5387861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.5395365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.5396054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.5428410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-08-17T12:37:51.5934587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.5935305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.5939186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.5939891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.6127166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:51.6127717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:51.6129649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:51.6130212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:51.6612000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot5pkbmg 2022-08-17T12:37:51.6613168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot5pkbmg/_remote_module_non_scriptable.py 2022-08-17T12:37:51.6941122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:51.7127197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bgfow5p 2022-08-17T12:37:51.7130436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bgfow5p/_remote_module_non_scriptable.py 2022-08-17T12:37:51.7733000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zt5g_ni 2022-08-17T12:37:51.7734916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zt5g_ni/_remote_module_non_scriptable.py 2022-08-17T12:37:51.8068173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25w8io1s 2022-08-17T12:37:51.8069540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25w8io1s/_remote_module_non_scriptable.py 2022-08-17T12:37:51.9625351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:51.9816732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:52.0420021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:52.0947421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-08-17T12:37:52.3653460Z 2022-08-17 12:37:52,364 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-08-17T12:37:52.3659349Z 2022-08-17 12:37:52,364 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-08-17T12:37:52.3754696Z 2022-08-17 12:37:52,374 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-08-17T12:37:52.3755977Z 2022-08-17 12:37:52,374 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-08-17T12:37:52.3857955Z 2022-08-17 12:37:52,384 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-08-17T12:37:52.3859346Z 2022-08-17 12:37:52,385 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-08-17T12:37:52.3953045Z 2022-08-17 12:37:52,394 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-08-17T12:37:52.3954180Z 2022-08-17 12:37:52,394 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-08-17T12:37:52.3962608Z 2022-08-17 12:37:52,395 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-08-17T12:37:52.3971401Z 2022-08-17 12:37:52,396 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-08-17T12:37:52.4165716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:52.4170281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-08-17T12:37:52.4175197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-08-17T12:37:52.4180806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:52.4188736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:52.4190204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:52.4191502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4194719Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4274291Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4278149Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4283823Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4285523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:52.4289545Z 2022-08-17 12:37:52,428 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-08-17T12:37:52.4292281Z 2022-08-17 12:37:52,428 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-08-17T12:37:52.4297722Z 2022-08-17 12:37:52,429 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-08-17T12:37:52.4304206Z 2022-08-17 12:37:52,429 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-08-17T12:37:52.4384579Z 2022-08-17 12:37:52,437 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-08-17T12:37:52.4388578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-08-17T12:37:52.4389566Z 2022-08-17 12:37:52,438 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers. 2022-08-17T12:37:52.4399831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-08-17T12:37:52.4609989Z 2022-08-17 12:37:52,460 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3 2022-08-17T12:37:52.4615808Z 2022-08-17 12:37:52,461 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3 2022-08-17T12:37:52.6010216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T12:37:52.6110653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T12:37:52.6212326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-08-17T12:37:52.6213249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T12:37:52.6218389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6223144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6227530Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6231724Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6237903Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6253540Z 2022-08-17 12:37:52,624 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:52.6259587Z 2022-08-17 12:37:52,625 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:52.6269466Z 2022-08-17 12:37:52,626 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:52.6273549Z 2022-08-17 12:37:52,626 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:52.6309289Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-08-17T12:37:52.6336694Z 2022-08-17 12:37:52,633 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:52.6348735Z 2022-08-17 12:37:52,634 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:52.6349925Z 2022-08-17 12:37:52,634 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:52.6369161Z 2022-08-17 12:37:52,636 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP. 2022-08-17T12:37:52.6546929Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:52.6548069Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:52.6706459Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:52.6707600Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:52.6810496Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:52.6811568Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:52.6925255Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:627: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it. 2022-08-17T12:37:52.6926436Z "The `check_reduction` argument in `DistributedDataParallel` " 2022-08-17T12:37:52.7001054Z 2022-08-17 12:37:52,699 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:52.7006257Z 2022-08-17 12:37:52,699 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:52.7016067Z 2022-08-17 12:37:52,701 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2 2022-08-17T12:37:52.7016766Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.7047465Z 2022-08-17 12:37:52,704 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:52.7071790Z 2022-08-17 12:37:52,706 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params. 2022-08-17T12:37:52.7076231Z 2022-08-17 12:37:52,707 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2 2022-08-17T12:37:52.7076919Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.7814827Z 2022-08-17 12:37:52,780 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:52.7815319Z [-1., -1.], 2022-08-17T12:37:52.7815585Z [-1., 1.], 2022-08-17T12:37:52.7816028Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7816651Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.7816948Z tensor([[-1., 1.], 2022-08-17T12:37:52.7817177Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.7817378Z [ 0., 4.]])} 2022-08-17T12:37:52.7817846Z 2022-08-17 12:37:52,780 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:52.7818194Z [ 1., -1.], 2022-08-17T12:37:52.7818361Z [ 1., 1.], 2022-08-17T12:37:52.7818789Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7819243Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.7819763Z tensor([[-1., 1.], 2022-08-17T12:37:52.7819983Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.7820197Z [ 0., 4.]])} 2022-08-17T12:37:52.7820659Z 2022-08-17 12:37:52,780 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.7820992Z [-1., 1.], 2022-08-17T12:37:52.7821198Z [-1., -1.], 2022-08-17T12:37:52.7821613Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7822059Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.7822335Z tensor([[-1., 1.], 2022-08-17T12:37:52.7822561Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.7822777Z [ 0., 4.]])} 2022-08-17T12:37:52.7823222Z 2022-08-17 12:37:52,780 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.7823533Z [ 1., 1.], 2022-08-17T12:37:52.7823737Z [ 1., -1.], 2022-08-17T12:37:52.7824129Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7824573Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.7824854Z tensor([[-1., 1.], 2022-08-17T12:37:52.7825083Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.7825286Z [ 0., 4.]])} 2022-08-17T12:37:52.7839850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:52.7840697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T12:37:52.7975214Z 2022-08-17 12:37:52,796 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.7976049Z [-1., 1.], 2022-08-17T12:37:52.7976521Z [-1., -1.], 2022-08-17T12:37:52.7977452Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7978419Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.7979046Z tensor([[-1., 1.], 2022-08-17T12:37:52.7979673Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.7980186Z [-2., 2.]])} 2022-08-17T12:37:52.7981598Z 2022-08-17 12:37:52,797 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.7982280Z [ 1., 1.], 2022-08-17T12:37:52.7982722Z [ 1., -1.], 2022-08-17T12:37:52.7983456Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.7984317Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.7984926Z tensor([[-1., 1.], 2022-08-17T12:37:52.7985526Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.7986062Z [-2., 2.]])} 2022-08-17T12:37:52.8035095Z 2022-08-17 12:37:52,802 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2 2022-08-17T12:37:52.8035850Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.8054924Z 2022-08-17 12:37:52,804 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2 2022-08-17T12:37:52.8055682Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.8218189Z 2022-08-17 12:37:52,821 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:52.8218883Z [-1., -1.], 2022-08-17T12:37:52.8219275Z [-1., 1.], 2022-08-17T12:37:52.8220056Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8220849Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8221340Z tensor([[-1., 1.], 2022-08-17T12:37:52.8221750Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8222135Z [ 0., 4.]])} 2022-08-17T12:37:52.8222963Z 2022-08-17 12:37:52,821 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:52.8223569Z [ 1., -1.], 2022-08-17T12:37:52.8223828Z [ 1., 1.], 2022-08-17T12:37:52.8224525Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8225423Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8225957Z tensor([[-1., 1.], 2022-08-17T12:37:52.8226451Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8226900Z [ 0., 4.]])} 2022-08-17T12:37:52.8228005Z 2022-08-17 12:37:52,821 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.8228681Z [ 1., 1.], 2022-08-17T12:37:52.8229133Z [ 1., -1.], 2022-08-17T12:37:52.8230017Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8230984Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8231596Z tensor([[-1., 1.], 2022-08-17T12:37:52.8232061Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8232517Z [ 0., 4.]])} 2022-08-17T12:37:52.8233415Z 2022-08-17 12:37:52,821 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.8233962Z [-1., 1.], 2022-08-17T12:37:52.8234254Z [-1., -1.], 2022-08-17T12:37:52.8234925Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8235654Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8236303Z tensor([[-1., 1.], 2022-08-17T12:37:52.8236667Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8237006Z [ 0., 4.]])} 2022-08-17T12:37:52.8326510Z 2022-08-17 12:37:52,832 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.8327184Z [-1., 1.], 2022-08-17T12:37:52.8327591Z [-1., -1.], 2022-08-17T12:37:52.8328487Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8329275Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.8329857Z tensor([[-1., 1.], 2022-08-17T12:37:52.8330400Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.8330900Z [-2., 2.]])} 2022-08-17T12:37:52.8337005Z 2022-08-17 12:37:52,833 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.8337591Z [ 1., 1.], 2022-08-17T12:37:52.8337974Z [ 1., -1.], 2022-08-17T12:37:52.8339106Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8340082Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.8340700Z tensor([[-1., 1.], 2022-08-17T12:37:52.8341319Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.8341999Z [-2., 2.]])} 2022-08-17T12:37:52.8379333Z 2022-08-17 12:37:52,837 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2 2022-08-17T12:37:52.8380079Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.8382277Z 2022-08-17 12:37:52,837 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2 2022-08-17T12:37:52.8383036Z to 1 to simulate uneven inputs. 2022-08-17T12:37:52.8522384Z 2022-08-17 12:37:52,851 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:52.8523021Z [ 1., -1.], 2022-08-17T12:37:52.8523316Z [ 1., 1.], 2022-08-17T12:37:52.8524086Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8524822Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8525294Z tensor([[-1., 1.], 2022-08-17T12:37:52.8525652Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8526211Z [ 0., 4.]])} 2022-08-17T12:37:52.8527021Z 2022-08-17 12:37:52,851 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.8527526Z [ 1., 1.], 2022-08-17T12:37:52.8527849Z [ 1., -1.], 2022-08-17T12:37:52.8528537Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8529305Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8529771Z tensor([[-1., 1.], 2022-08-17T12:37:52.8530113Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8530463Z [ 0., 4.]])} 2022-08-17T12:37:52.8531253Z 2022-08-17 12:37:52,851 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:52.8531836Z [-1., -1.], 2022-08-17T12:37:52.8532448Z [-1., 1.], 2022-08-17T12:37:52.8533138Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8533894Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8534373Z tensor([[-1., 1.], 2022-08-17T12:37:52.8534709Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8535026Z [ 0., 4.]])} 2022-08-17T12:37:52.8540404Z 2022-08-17 12:37:52,853 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.8541023Z [-1., 1.], 2022-08-17T12:37:52.8541401Z [-1., -1.], 2022-08-17T12:37:52.8542114Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8542889Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing: 2022-08-17T12:37:52.8543342Z tensor([[-1., 1.], 2022-08-17T12:37:52.8543701Z [ 1., 1.]], requires_grad=True): tensor([[ 0., 12.], 2022-08-17T12:37:52.8544029Z [ 0., 4.]])} 2022-08-17T12:37:52.8680147Z 2022-08-17 12:37:52,867 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:52.8680788Z [-1., 1.], 2022-08-17T12:37:52.8681114Z [-1., -1.], 2022-08-17T12:37:52.8681788Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8682629Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.8683242Z tensor([[-1., 1.], 2022-08-17T12:37:52.8683858Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.8684408Z [-2., 2.]])} 2022-08-17T12:37:52.8706359Z 2022-08-17 12:37:52,870 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:52.8706867Z [ 1., 1.], 2022-08-17T12:37:52.8707280Z [ 1., -1.], 2022-08-17T12:37:52.8707948Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:52.8708639Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing: 2022-08-17T12:37:52.8709051Z tensor([[-1., 1.], 2022-08-17T12:37:52.8709527Z [ 1., 1.]], requires_grad=True): tensor([[ 2., -2.], 2022-08-17T12:37:52.8710090Z [-2., 2.]])} 2022-08-17T12:37:52.8756001Z 2022-08-17 12:37:52,875 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-08-17T12:37:52.8762330Z 2022-08-17 12:37:52,875 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-08-17T12:37:52.8769849Z 2022-08-17 12:37:52,876 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-08-17T12:37:52.8776834Z 2022-08-17 12:37:52,877 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-08-17T12:37:52.8786051Z 2022-08-17 12:37:52,878 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-08-17T12:37:52.9648985Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:37:52.9649744Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:37:52.9720078Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:37:52.9720792Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:37:52.9735985Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:37:52.9736572Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:37:52.9741975Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-08-17T12:37:52.9742360Z _warnings.warn(warn_message, ResourceWarning) 2022-08-17T12:37:53.4356997Z ok (3.792s) 2022-08-17T12:37:53.4357218Z 2022-08-17T12:37:53.4357685Z ---------------------------------------------------------------------- 2022-08-17T12:37:53.4357941Z Ran 1 test in 3.793s 2022-08-17T12:37:53.4358067Z 2022-08-17T12:37:53.4358131Z OK 2022-08-17T12:37:53.4358229Z 2022-08-17T12:37:53.4358324Z Generating XML reports... 2022-08-17T12:37:53.4394379Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123749.xml 2022-08-17T12:37:54.7200253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:54.7200691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:54.7203365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:54.7203990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:54.8716010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s3ex_8h 2022-08-17T12:37:54.8719335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s3ex_8h/_remote_module_non_scriptable.py 2022-08-17T12:37:55.1357081Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:37:55.1367895Z 2022-08-17T12:37:55.1368003Z Running tests... 2022-08-17T12:37:55.1369035Z ---------------------------------------------------------------------- 2022-08-17T12:37:55.3998224Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3193 2022-08-17T12:37:55.4029298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3194 2022-08-17T12:37:55.4059109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3195 2022-08-17T12:37:55.4090923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3196 2022-08-17T12:37:55.4123904Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3197 2022-08-17T12:37:55.4157172Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3198 2022-08-17T12:37:56.7031816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:56.7032463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:56.7036414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:56.7037102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:56.7659565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:56.7660142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:56.7663714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:56.7664323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:56.7873793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:56.7874425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:56.7878894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:56.7879853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:56.8316277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:56.8316979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:56.8321409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:56.8637602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:56.8638288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyww_5dnb 2022-08-17T12:37:56.8639034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyww_5dnb/_remote_module_non_scriptable.py 2022-08-17T12:37:56.9273270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87kpydm9 2022-08-17T12:37:56.9274277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87kpydm9/_remote_module_non_scriptable.py 2022-08-17T12:37:57.0368901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_q8uw_a 2022-08-17T12:37:57.0369701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_q8uw_a/_remote_module_non_scriptable.py 2022-08-17T12:37:57.0777467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7nlt7u1 2022-08-17T12:37:57.0778845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7nlt7u1/_remote_module_non_scriptable.py 2022-08-17T12:37:57.0866591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:57.0867307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:57.0868060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:37:57.0868654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:37:57.0871788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:57.0872411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:57.0873274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:37:57.0873884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:37:57.1320273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-08-17T12:37:57.1990013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-08-17T12:37:57.3289486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3aiodv3 2022-08-17T12:37:57.3290324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5e6osa4_ 2022-08-17T12:37:57.3291131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3aiodv3/_remote_module_non_scriptable.py 2022-08-17T12:37:57.3291921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5e6osa4_/_remote_module_non_scriptable.py 2022-08-17T12:37:57.3915524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:37:57.4172688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:37:57.5873677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:37:57.5917015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:37:57.8745349Z 2022-08-17 12:37:57,873 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0... 2022-08-17T12:37:57.8746949Z 2022-08-17 12:37:57,873 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3] 2022-08-17T12:37:57.8939289Z 2022-08-17 12:37:57,893 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1... 2022-08-17T12:37:57.8940535Z 2022-08-17 12:37:57,893 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3] 2022-08-17T12:37:57.9039060Z 2022-08-17 12:37:57,903 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2... 2022-08-17T12:37:57.9040303Z 2022-08-17 12:37:57,903 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3] 2022-08-17T12:37:57.9153710Z 2022-08-17 12:37:57,913 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process... 2022-08-17T12:37:57.9155248Z 2022-08-17 12:37:57,913 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3... 2022-08-17T12:37:57.9156837Z 2022-08-17 12:37:57,913 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3] 2022-08-17T12:37:57.9158373Z 2022-08-17 12:37:57,914 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running. 2022-08-17T12:37:57.9274197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:37:57.9283458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:37:57.9284352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-08-17T12:37:57.9292724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-08-17T12:37:57.9307060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:37:57.9308690Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9309998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:37:57.9311332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9383567Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9389961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9391226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9395403Z 2022-08-17 12:37:57,938 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0... 2022-08-17T12:37:57.9399912Z 2022-08-17 12:37:57,939 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1... 2022-08-17T12:37:57.9404970Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-08-17T12:37:57.9410461Z 2022-08-17 12:37:57,940 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2... 2022-08-17T12:37:57.9415023Z 2022-08-17 12:37:57,941 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3... 2022-08-17T12:37:57.9492955Z 2022-08-17 12:37:57,948 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master 2022-08-17T12:37:57.9598252Z 2022-08-17 12:37:57,959 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:57.9599611Z 2022-08-17 12:37:57,959 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-08-17T12:37:57.9831774Z 2022-08-17 12:37:57,982 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-2: Initing RemoteNet with 5 3 2022-08-17T12:37:57.9840222Z 2022-08-17 12:37:57,983 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-3: Initing RemoteEM with 2 3 2022-08-17T12:37:58.0065964Z 2022-08-17 12:37:58,005 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:58.0066606Z [ 1., -1.], 2022-08-17T12:37:58.0066901Z [ 1., 1.], 2022-08-17T12:37:58.0067755Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.0068594Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.0069094Z tensor([[-1., 1.], 2022-08-17T12:37:58.0069627Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:58.0070234Z [ 0., -16.]])} 2022-08-17T12:37:58.0487930Z 2022-08-17 12:37:58,047 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:58.0489429Z 2022-08-17 12:37:58,048 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-08-17T12:37:58.0505443Z 2022-08-17 12:37:58,049 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:58.0506618Z 2022-08-17 12:37:58,050 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-08-17T12:37:58.0793350Z 2022-08-17 12:37:58,078 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:58.0793955Z [ 1., 1.], 2022-08-17T12:37:58.0794352Z [ 1., -1.], 2022-08-17T12:37:58.0795119Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.0795804Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.0796126Z tensor([[-1., 1.], 2022-08-17T12:37:58.0796427Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:58.0796724Z [ 0., 16.]])} 2022-08-17T12:37:58.0797393Z 2022-08-17 12:37:58,078 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:58.0797902Z [-1., -1.], 2022-08-17T12:37:58.0798194Z [-1., 1.], 2022-08-17T12:37:58.0798772Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.0799233Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.0799524Z tensor([[-1., 1.], 2022-08-17T12:37:58.0799813Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:58.0800054Z [ 16., -16.]])} 2022-08-17T12:37:58.0907249Z 2022-08-17 12:37:58,090 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters. 2022-08-17T12:37:58.0908311Z 2022-08-17 12:37:58,090 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params. 2022-08-17T12:37:58.1105460Z 2022-08-17 12:37:58,109 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:58.1106153Z [-1., 1.], 2022-08-17T12:37:58.1106578Z [-1., -1.], 2022-08-17T12:37:58.1107981Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1108961Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.1109575Z tensor([[-1., 1.], 2022-08-17T12:37:58.1110190Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:58.1110728Z [-16., 16.]])} 2022-08-17T12:37:58.1283689Z 2022-08-17 12:37:58,127 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:58.1284441Z [ 1., 1.], 2022-08-17T12:37:58.1284887Z [ 1., -1.], 2022-08-17T12:37:58.1285814Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1287056Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.1289408Z tensor([[-1., 1.], 2022-08-17T12:37:58.1294255Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:58.1297582Z [ 0., 16.]])} 2022-08-17T12:37:58.1302690Z 2022-08-17 12:37:58,127 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:58.1307569Z [ 1., -1.], 2022-08-17T12:37:58.1312012Z [ 1., 1.], 2022-08-17T12:37:58.1315849Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1319757Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.1324078Z tensor([[-1., 1.], 2022-08-17T12:37:58.1327749Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:58.1331578Z [ 0., -16.]])} 2022-08-17T12:37:58.1335856Z 2022-08-17 12:37:58,128 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:58.1339754Z [-1., -1.], 2022-08-17T12:37:58.1343239Z [-1., 1.], 2022-08-17T12:37:58.1345503Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1346856Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.1347870Z tensor([[-1., 1.], 2022-08-17T12:37:58.1348883Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:58.1349714Z [ 16., -16.]])} 2022-08-17T12:37:58.1350888Z 2022-08-17 12:37:58,128 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:58.1351493Z [-1., 1.], 2022-08-17T12:37:58.1351818Z [-1., -1.], 2022-08-17T12:37:58.1352481Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1353304Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.1353837Z tensor([[-1., 1.], 2022-08-17T12:37:58.1354374Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:58.1354901Z [-16., 16.]])} 2022-08-17T12:37:58.1470842Z 2022-08-17 12:37:58,146 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.], 2022-08-17T12:37:58.1472442Z [ 1., -1.], 2022-08-17T12:37:58.1473014Z [ 1., 1.], 2022-08-17T12:37:58.1475591Z [ 1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1476689Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.1477214Z tensor([[-1., 1.], 2022-08-17T12:37:58.1477742Z [ 1., 1.]], requires_grad=True): tensor([[-32., -16.], 2022-08-17T12:37:58.1478161Z [ 0., -16.]])} 2022-08-17T12:37:58.1478900Z 2022-08-17 12:37:58,146 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.], 2022-08-17T12:37:58.1479418Z [-1., -1.], 2022-08-17T12:37:58.1479726Z [-1., 1.], 2022-08-17T12:37:58.1480505Z [-1., 1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1481219Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., -28., -28.]]), Parameter containing: 2022-08-17T12:37:58.1481620Z tensor([[-1., 1.], 2022-08-17T12:37:58.1482203Z [ 1., 1.]], requires_grad=True): tensor([[ 16., -16.], 2022-08-17T12:37:58.1482763Z [ 16., -16.]])} 2022-08-17T12:37:58.1504049Z 2022-08-17 12:37:58,149 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., 1.], 2022-08-17T12:37:58.1504582Z [ 1., 1.], 2022-08-17T12:37:58.1505198Z [ 1., -1.], 2022-08-17T12:37:58.1505955Z [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1506733Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.1507340Z tensor([[-1., 1.], 2022-08-17T12:37:58.1507792Z [ 1., 1.]], requires_grad=True): tensor([[32., 16.], 2022-08-17T12:37:58.1508248Z [ 0., 16.]])} 2022-08-17T12:37:58.1524661Z 2022-08-17 12:37:58,152 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., 1.], 2022-08-17T12:37:58.1525296Z [-1., 1.], 2022-08-17T12:37:58.1525787Z [-1., -1.], 2022-08-17T12:37:58.1526733Z [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing: 2022-08-17T12:37:58.1527548Z tensor([[-1., 1., 1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing: 2022-08-17T12:37:58.1528164Z tensor([[-1., 1.], 2022-08-17T12:37:58.1528788Z [ 1., 1.]], requires_grad=True): tensor([[-16., 16.], 2022-08-17T12:37:58.1529306Z [-16., 16.]])} 2022-08-17T12:37:58.1584950Z 2022-08-17 12:37:58,158 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0... 2022-08-17T12:37:58.1589247Z 2022-08-17 12:37:58,158 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1... 2022-08-17T12:37:58.1595778Z 2022-08-17 12:37:58,159 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2... 2022-08-17T12:37:58.1603999Z 2022-08-17 12:37:58,160 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3... 2022-08-17T12:37:58.1614196Z 2022-08-17 12:37:58,161 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker. 2022-08-17T12:37:58.7252889Z ok (3.588s) 2022-08-17T12:37:58.7253161Z 2022-08-17T12:37:58.7253708Z ---------------------------------------------------------------------- 2022-08-17T12:37:58.7254030Z Ran 1 test in 3.588s 2022-08-17T12:37:58.7254155Z 2022-08-17T12:37:58.7254205Z OK 2022-08-17T12:37:58.7254300Z 2022-08-17T12:37:58.7254394Z Generating XML reports... 2022-08-17T12:37:58.7287402Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123755.xml 2022-08-17T12:38:00.0312333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:00.0313028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:00.0315294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:00.0315673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:00.1839607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnm6j5ni 2022-08-17T12:38:00.1842651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnm6j5ni/_remote_module_non_scriptable.py 2022-08-17T12:38:00.4486066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:00.4496909Z 2022-08-17T12:38:00.4497088Z Running tests... 2022-08-17T12:38:00.4497468Z ---------------------------------------------------------------------- 2022-08-17T12:38:00.4503660Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:38:00.7136793Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3589 2022-08-17T12:38:00.7166921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3590 2022-08-17T12:38:00.7196103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3591 2022-08-17T12:38:00.7228221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3592 2022-08-17T12:38:01.7511669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:01.7512313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:01.7515054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:01.7515717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:01.7599915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:01.7600614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:01.7604692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:01.7605402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:01.7722162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:01.7722814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:01.7726341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:01.7726987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:01.7797816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:01.7798511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:01.7802724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:01.7803434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:01.9117536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y9vel_o 2022-08-17T12:38:01.9118399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y9vel_o/_remote_module_non_scriptable.py 2022-08-17T12:38:01.9218359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtlk84fo 2022-08-17T12:38:01.9219624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtlk84fo/_remote_module_non_scriptable.py 2022-08-17T12:38:01.9324759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1phogu0f 2022-08-17T12:38:01.9326306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1phogu0f/_remote_module_non_scriptable.py 2022-08-17T12:38:01.9406303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1dwtg8t 2022-08-17T12:38:01.9407271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1dwtg8t/_remote_module_non_scriptable.py 2022-08-17T12:38:02.1713140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:02.1817529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:02.1935647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:02.2043083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:02.4639900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:38:02.4740459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:38:02.4742473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:38:02.4743908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:02.4748869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:38:02.4750186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:02.4751429Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:02.4844525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:03.0290007Z ok (2.579s) 2022-08-17T12:38:03.0290324Z 2022-08-17T12:38:03.0290755Z ---------------------------------------------------------------------- 2022-08-17T12:38:03.0291081Z Ran 1 test in 2.579s 2022-08-17T12:38:03.0291227Z 2022-08-17T12:38:03.0291294Z OK 2022-08-17T12:38:03.0291376Z 2022-08-17T12:38:03.0291472Z Generating XML reports... 2022-08-17T12:38:03.0324751Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123800.xml 2022-08-17T12:38:04.3381117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:04.3381515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:04.3384175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:04.3384554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:04.4909147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekifzeo2 2022-08-17T12:38:04.4913057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekifzeo2/_remote_module_non_scriptable.py 2022-08-17T12:38:04.7575765Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:04.7599297Z 2022-08-17T12:38:04.7599700Z Running tests... 2022-08-17T12:38:04.7600408Z ---------------------------------------------------------------------- 2022-08-17T12:38:05.0219218Z test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3904 2022-08-17T12:38:05.0248632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3905 2022-08-17T12:38:05.0278229Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3906 2022-08-17T12:38:05.0310862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3907 2022-08-17T12:38:06.0515300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:06.0516252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:06.0519190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:06.0519857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:06.0826348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:06.0827042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:06.0830118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:06.0830759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:06.0871086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:06.0871772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:06.0874910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:06.0875888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:06.1018178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:06.1018858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:06.1022127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:06.1022793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:06.2104714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj7rlvo5 2022-08-17T12:38:06.2105510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj7rlvo5/_remote_module_non_scriptable.py 2022-08-17T12:38:06.2422042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpnsadbc 2022-08-17T12:38:06.2423202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpnsadbc/_remote_module_non_scriptable.py 2022-08-17T12:38:06.2461965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m6i5wy6 2022-08-17T12:38:06.2464217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m6i5wy6/_remote_module_non_scriptable.py 2022-08-17T12:38:06.2589457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39sye8it 2022-08-17T12:38:06.2590294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39sye8it/_remote_module_non_scriptable.py 2022-08-17T12:38:06.4734858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:06.5070570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:06.5071246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:06.5226414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:07.7381622Z ok (2.979s) 2022-08-17T12:38:07.7381924Z 2022-08-17T12:38:07.7382444Z ---------------------------------------------------------------------- 2022-08-17T12:38:07.7382715Z Ran 1 test in 2.979s 2022-08-17T12:38:07.7382838Z 2022-08-17T12:38:07.7382905Z OK 2022-08-17T12:38:07.7383003Z 2022-08-17T12:38:07.7383088Z Generating XML reports... 2022-08-17T12:38:07.7416738Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123804.xml 2022-08-17T12:38:09.0696880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:09.0697269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:09.0699799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:09.0700176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:09.2224999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxys7d3b9 2022-08-17T12:38:09.2228786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxys7d3b9/_remote_module_non_scriptable.py 2022-08-17T12:38:09.4865380Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:09.4876061Z 2022-08-17T12:38:09.4876206Z Running tests... 2022-08-17T12:38:09.4876670Z ---------------------------------------------------------------------- 2022-08-17T12:38:09.7508162Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4127 2022-08-17T12:38:09.7537108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4128 2022-08-17T12:38:09.7565554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4129 2022-08-17T12:38:09.7595697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4130 2022-08-17T12:38:10.7745057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:10.7745807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:10.7747586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:10.7748207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:10.7825526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:10.7829847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:10.7830792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:10.7831419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:10.8000751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:10.8001380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:10.8004154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:10.8004739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:10.8011859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:10.8012653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:10.8016239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:10.8016884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:10.9325565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2vb_gsm 2022-08-17T12:38:10.9326629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2vb_gsm/_remote_module_non_scriptable.py 2022-08-17T12:38:10.9423097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftaish_1 2022-08-17T12:38:10.9424480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftaish_1/_remote_module_non_scriptable.py 2022-08-17T12:38:10.9588899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrv6vato 2022-08-17T12:38:10.9590207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrv6vato/_remote_module_non_scriptable.py 2022-08-17T12:38:10.9603964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98bvgzwr 2022-08-17T12:38:10.9606043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98bvgzwr/_remote_module_non_scriptable.py 2022-08-17T12:38:11.1928359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:11.2009955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:11.2166167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:11.2211055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:11.8656143Z ok (2.378s) 2022-08-17T12:38:11.8656320Z 2022-08-17T12:38:11.8656757Z ---------------------------------------------------------------------- 2022-08-17T12:38:11.8657027Z Ran 1 test in 2.378s 2022-08-17T12:38:11.8657151Z 2022-08-17T12:38:11.8657222Z OK 2022-08-17T12:38:11.8657337Z 2022-08-17T12:38:11.8657432Z Generating XML reports... 2022-08-17T12:38:11.8690638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123809.xml 2022-08-17T12:38:13.1435813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:13.1436475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:13.1438074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:13.1438685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:13.2917664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztnt_cfy 2022-08-17T12:38:13.2921498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztnt_cfy/_remote_module_non_scriptable.py 2022-08-17T12:38:13.5520149Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:13.5531414Z 2022-08-17T12:38:13.5531901Z Running tests... 2022-08-17T12:38:13.5532342Z ---------------------------------------------------------------------- 2022-08-17T12:38:13.8117820Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4370 2022-08-17T12:38:13.8145946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4371 2022-08-17T12:38:13.8175690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4372 2022-08-17T12:38:13.8207153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4373 2022-08-17T12:38:14.8442324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:14.8442954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:14.8446486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:14.8447150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:14.8567184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:14.8567858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:14.8571645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:14.8572258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:14.8631078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:14.8631707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:14.8634250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:14.8635172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:14.8796888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:14.8797532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:14.8801103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:14.8801755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:15.0043978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa3_34a2 2022-08-17T12:38:15.0044839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa3_34a2/_remote_module_non_scriptable.py 2022-08-17T12:38:15.0246840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyb9w31f 2022-08-17T12:38:15.0247628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyb9w31f/_remote_module_non_scriptable.py 2022-08-17T12:38:15.0333466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjchrjhi 2022-08-17T12:38:15.0334626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjchrjhi/_remote_module_non_scriptable.py 2022-08-17T12:38:15.0434227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2welp12i 2022-08-17T12:38:15.0435416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2welp12i/_remote_module_non_scriptable.py 2022-08-17T12:38:15.2632117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:15.2842618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:15.2905785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:15.3016616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:16.0268364Z ok (2.473s) 2022-08-17T12:38:16.0268762Z 2022-08-17T12:38:16.0269305Z ---------------------------------------------------------------------- 2022-08-17T12:38:16.0269715Z Ran 1 test in 2.474s 2022-08-17T12:38:16.0269824Z 2022-08-17T12:38:16.0269907Z OK 2022-08-17T12:38:16.0270004Z 2022-08-17T12:38:16.0270102Z Generating XML reports... 2022-08-17T12:38:16.0302928Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123813.xml 2022-08-17T12:38:17.3321540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:17.3322224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:17.3324833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:17.3325423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:17.4829129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkk06te3e 2022-08-17T12:38:17.4832672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkk06te3e/_remote_module_non_scriptable.py 2022-08-17T12:38:17.7465437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:17.7476571Z 2022-08-17T12:38:17.7476882Z Running tests... 2022-08-17T12:38:17.7477561Z ---------------------------------------------------------------------- 2022-08-17T12:38:18.0076765Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4613 2022-08-17T12:38:18.0103680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4614 2022-08-17T12:38:18.0132139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4615 2022-08-17T12:38:18.0162133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4616 2022-08-17T12:38:19.0540082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:19.0540725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:19.0543940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:19.0544601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:19.1514327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:19.1515015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:19.1519395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:19.1520105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:19.1561392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:19.1562099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:19.1566815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:19.1567482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:19.1599910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:19.1600564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:19.1604572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:19.1605220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:19.2144871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkjo4w1j 2022-08-17T12:38:19.2145530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkjo4w1j/_remote_module_non_scriptable.py 2022-08-17T12:38:19.3124866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b6solp3 2022-08-17T12:38:19.3125709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b6solp3/_remote_module_non_scriptable.py 2022-08-17T12:38:19.3190732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24f6bwnn 2022-08-17T12:38:19.3192481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24f6bwnn/_remote_module_non_scriptable.py 2022-08-17T12:38:19.3208401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzc53jxyd 2022-08-17T12:38:19.3211049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzc53jxyd/_remote_module_non_scriptable.py 2022-08-17T12:38:19.4807661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:19.5767170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:19.5877642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:19.5881939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:20.3223872Z ok (2.574s) 2022-08-17T12:38:20.3224140Z 2022-08-17T12:38:20.3224597Z ---------------------------------------------------------------------- 2022-08-17T12:38:20.3225103Z Ran 1 test in 2.575s 2022-08-17T12:38:20.3225252Z 2022-08-17T12:38:20.3225316Z OK 2022-08-17T12:38:20.3225399Z 2022-08-17T12:38:20.3225496Z Generating XML reports... 2022-08-17T12:38:20.3258087Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123817.xml 2022-08-17T12:38:21.6343360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:21.6344004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:21.6346157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:21.6346853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:21.7861684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjmuh7zp 2022-08-17T12:38:21.7864458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjmuh7zp/_remote_module_non_scriptable.py 2022-08-17T12:38:22.0503573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:22.0514462Z 2022-08-17T12:38:22.0514589Z Running tests... 2022-08-17T12:38:22.0515219Z ---------------------------------------------------------------------- 2022-08-17T12:38:22.3189426Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4856 2022-08-17T12:38:22.3218288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4857 2022-08-17T12:38:22.3248001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4858 2022-08-17T12:38:22.3278532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4859 2022-08-17T12:38:23.3991099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:23.3991758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:23.3994486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:23.3995317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:23.4090479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:23.4091184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:23.4095288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:23.4107726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:23.4447363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:23.4448030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:23.4451447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:23.4452014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:23.4467825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:23.4468433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:23.4473157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:23.4473959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:23.5567219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeoyauybr 2022-08-17T12:38:23.5568140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeoyauybr/_remote_module_non_scriptable.py 2022-08-17T12:38:23.5678461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4baqwg39 2022-08-17T12:38:23.5680028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4baqwg39/_remote_module_non_scriptable.py 2022-08-17T12:38:23.6034963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h52c2k3 2022-08-17T12:38:23.6035992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h52c2k3/_remote_module_non_scriptable.py 2022-08-17T12:38:23.6046271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebts6qbg 2022-08-17T12:38:23.6048260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebts6qbg/_remote_module_non_scriptable.py 2022-08-17T12:38:23.8173746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:23.8302409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:23.8669529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:23.8683530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:24.5339031Z ok (2.482s) 2022-08-17T12:38:24.5339250Z 2022-08-17T12:38:24.5339666Z ---------------------------------------------------------------------- 2022-08-17T12:38:24.5339937Z Ran 1 test in 2.482s 2022-08-17T12:38:24.5340058Z 2022-08-17T12:38:24.5340124Z OK 2022-08-17T12:38:24.5340220Z 2022-08-17T12:38:24.5340319Z Generating XML reports... 2022-08-17T12:38:24.5373892Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123822.xml 2022-08-17T12:38:25.8335688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:25.8336269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:25.8342371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:25.8342864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:25.9857119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfwwpaj7 2022-08-17T12:38:25.9859993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfwwpaj7/_remote_module_non_scriptable.py 2022-08-17T12:38:26.2521582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:26.2532019Z 2022-08-17T12:38:26.2532163Z Running tests... 2022-08-17T12:38:26.2533378Z ---------------------------------------------------------------------- 2022-08-17T12:38:26.5232805Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5099 2022-08-17T12:38:26.5263225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5100 2022-08-17T12:38:26.5293573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5101 2022-08-17T12:38:26.5326962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5102 2022-08-17T12:38:27.5990191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:27.5990829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:27.5994223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:27.5994922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:27.6358527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:27.6359083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:27.6362633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:27.6363280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:27.6621859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:27.6622824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:27.6625396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:27.6626033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:27.6797167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:27.6797744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:27.6800570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:27.6801173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:27.7592128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp262xagjd 2022-08-17T12:38:27.7592902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp262xagjd/_remote_module_non_scriptable.py 2022-08-17T12:38:27.7954612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq9kvvbz 2022-08-17T12:38:27.7955416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq9kvvbz/_remote_module_non_scriptable.py 2022-08-17T12:38:27.8214222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf54lrntb 2022-08-17T12:38:27.8215077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf54lrntb/_remote_module_non_scriptable.py 2022-08-17T12:38:27.8383211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuvpi_7i 2022-08-17T12:38:27.8383981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuvpi_7i/_remote_module_non_scriptable.py 2022-08-17T12:38:28.0251114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:28.0595103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:28.0841774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:28.1010762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:28.8388490Z ok (2.585s) 2022-08-17T12:38:28.8388767Z 2022-08-17T12:38:28.8389335Z ---------------------------------------------------------------------- 2022-08-17T12:38:28.8389634Z Ran 1 test in 2.586s 2022-08-17T12:38:28.8389758Z 2022-08-17T12:38:28.8389824Z OK 2022-08-17T12:38:28.8389907Z 2022-08-17T12:38:28.8390002Z Generating XML reports... 2022-08-17T12:38:28.8422993Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123826.xml 2022-08-17T12:38:30.1313644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:30.1314049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:30.1316022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:30.1316482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:30.2839853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvga71cg1 2022-08-17T12:38:30.2843293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvga71cg1/_remote_module_non_scriptable.py 2022-08-17T12:38:30.5475346Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:30.5486285Z 2022-08-17T12:38:30.5486381Z Running tests... 2022-08-17T12:38:30.5487288Z ---------------------------------------------------------------------- 2022-08-17T12:38:30.8195998Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5342 2022-08-17T12:38:30.8226097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5343 2022-08-17T12:38:30.8254849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5344 2022-08-17T12:38:30.8284726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5345 2022-08-17T12:38:31.8803354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:31.8803990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:31.8807023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:31.8807660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:31.8844795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:31.8845423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:31.8848975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:31.8944054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:31.8945275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:31.8945907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:31.8948550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:31.8949202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:31.9217813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:31.9218526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:31.9221256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:31.9222073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:32.0403715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp127x363u 2022-08-17T12:38:32.0455061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp127x363u/_remote_module_non_scriptable.py 2022-08-17T12:38:32.0455802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo67jsq7s 2022-08-17T12:38:32.0457059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo67jsq7s/_remote_module_non_scriptable.py 2022-08-17T12:38:32.0560525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqkvk7v1 2022-08-17T12:38:32.0561791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqkvk7v1/_remote_module_non_scriptable.py 2022-08-17T12:38:32.0811515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpean4zvs7 2022-08-17T12:38:32.0812417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpean4zvs7/_remote_module_non_scriptable.py 2022-08-17T12:38:32.3023213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:32.3107907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:32.3200541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:32.3447133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:33.0344480Z ok (2.485s) 2022-08-17T12:38:33.0344697Z 2022-08-17T12:38:33.0345192Z ---------------------------------------------------------------------- 2022-08-17T12:38:33.0345570Z Ran 1 test in 2.486s 2022-08-17T12:38:33.0345795Z 2022-08-17T12:38:33.0345914Z OK 2022-08-17T12:38:33.0346090Z 2022-08-17T12:38:33.0346514Z Generating XML reports... 2022-08-17T12:38:33.0379873Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123830.xml 2022-08-17T12:38:34.3435101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:34.3435488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:34.3437997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:34.3438375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:34.4939509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplh8e0w_2 2022-08-17T12:38:34.4942816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplh8e0w_2/_remote_module_non_scriptable.py 2022-08-17T12:38:34.7567297Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:34.7577768Z 2022-08-17T12:38:34.7577883Z Running tests... 2022-08-17T12:38:34.7578388Z ---------------------------------------------------------------------- 2022-08-17T12:38:35.0171808Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s) 2022-08-17T12:38:35.0172390Z 2022-08-17T12:38:35.0172701Z ---------------------------------------------------------------------- 2022-08-17T12:38:35.0172952Z Ran 1 test in 0.259s 2022-08-17T12:38:35.0173076Z 2022-08-17T12:38:35.0173155Z OK (skipped=1) 2022-08-17T12:38:35.0173270Z 2022-08-17T12:38:35.0173432Z Generating XML reports... 2022-08-17T12:38:35.0196507Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123834.xml 2022-08-17T12:38:36.2674281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:36.2674748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:36.2676856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:36.2677268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:36.4222011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qw9o9nf 2022-08-17T12:38:36.4225506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qw9o9nf/_remote_module_non_scriptable.py 2022-08-17T12:38:36.6891697Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:36.6902322Z 2022-08-17T12:38:36.6902442Z Running tests... 2022-08-17T12:38:36.6902860Z ---------------------------------------------------------------------- 2022-08-17T12:38:36.9560510Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5579 2022-08-17T12:38:36.9592233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5580 2022-08-17T12:38:36.9621222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5581 2022-08-17T12:38:36.9652566Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5582 2022-08-17T12:38:38.0230914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:38.0231573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:38.0234566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:38.0291819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:38.0292846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:38.0293492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:38.0297069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:38.0297713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:38.1075557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:38.1076249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:38.1079647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:38.1080290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:38.1323404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:38.1324113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:38.1327571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:38.1328264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:38.1825374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fq6c6sw 2022-08-17T12:38:38.1826271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fq6c6sw/_remote_module_non_scriptable.py 2022-08-17T12:38:38.1857195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufqjo7yz 2022-08-17T12:38:38.1858139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufqjo7yz/_remote_module_non_scriptable.py 2022-08-17T12:38:38.2627855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm2widmj 2022-08-17T12:38:38.2628651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm2widmj/_remote_module_non_scriptable.py 2022-08-17T12:38:38.2876711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrgh6fm3 2022-08-17T12:38:38.2878777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrgh6fm3/_remote_module_non_scriptable.py 2022-08-17T12:38:38.4427873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:38.4428311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:38.5215832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:38.5461358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:39.2715636Z ok (2.581s) 2022-08-17T12:38:39.2715889Z 2022-08-17T12:38:39.2716216Z ---------------------------------------------------------------------- 2022-08-17T12:38:39.2716485Z Ran 1 test in 2.581s 2022-08-17T12:38:39.2716610Z 2022-08-17T12:38:39.2716660Z OK 2022-08-17T12:38:39.2716758Z 2022-08-17T12:38:39.2749164Z Generating XML reports... 2022-08-17T12:38:39.2749861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123836.xml 2022-08-17T12:38:40.5665192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:40.5665587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:40.5668468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:40.5668883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:40.7194204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8u3e577 2022-08-17T12:38:40.7197139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8u3e577/_remote_module_non_scriptable.py 2022-08-17T12:38:40.9838597Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:40.9848778Z 2022-08-17T12:38:40.9848915Z Running tests... 2022-08-17T12:38:40.9849522Z ---------------------------------------------------------------------- 2022-08-17T12:38:41.2472968Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s) 2022-08-17T12:38:41.2473503Z 2022-08-17T12:38:41.2473720Z ---------------------------------------------------------------------- 2022-08-17T12:38:41.2474002Z Ran 1 test in 0.262s 2022-08-17T12:38:41.2474124Z 2022-08-17T12:38:41.2474200Z OK (skipped=1) 2022-08-17T12:38:41.2474301Z 2022-08-17T12:38:41.2474391Z Generating XML reports... 2022-08-17T12:38:41.2501956Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123840.xml 2022-08-17T12:38:42.5068941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:42.5069361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:42.5071806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:42.5072184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:42.6575490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlwq2i23 2022-08-17T12:38:42.6578774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlwq2i23/_remote_module_non_scriptable.py 2022-08-17T12:38:42.9214553Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:42.9225888Z 2022-08-17T12:38:42.9226102Z Running tests... 2022-08-17T12:38:42.9226801Z ---------------------------------------------------------------------- 2022-08-17T12:38:43.1874399Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5832 2022-08-17T12:38:43.1901641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5833 2022-08-17T12:38:43.1929444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5834 2022-08-17T12:38:43.1959846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5835 2022-08-17T12:38:44.2713162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:44.2713846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:44.2715377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:44.2715984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:44.2727159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:44.2727744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:44.2732178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:44.2732817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:44.2821923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:44.2822920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:44.2826853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:44.2827484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:44.3040316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:44.3041027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:44.3044562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:44.3045148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:44.4319425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidp55zor 2022-08-17T12:38:44.4320398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidp55zor/_remote_module_non_scriptable.py 2022-08-17T12:38:44.4327398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuagzckb 2022-08-17T12:38:44.4329774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuagzckb/_remote_module_non_scriptable.py 2022-08-17T12:38:44.4411824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkzkl7ju 2022-08-17T12:38:44.4413153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkzkl7ju/_remote_module_non_scriptable.py 2022-08-17T12:38:44.4635149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5j51564w 2022-08-17T12:38:44.4635884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5j51564w/_remote_module_non_scriptable.py 2022-08-17T12:38:44.6939417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:44.6961932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:44.7019258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:44.7274225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:45.4022734Z ok (2.479s) 2022-08-17T12:38:45.4023000Z 2022-08-17T12:38:45.4023430Z ---------------------------------------------------------------------- 2022-08-17T12:38:45.4023688Z Ran 1 test in 2.480s 2022-08-17T12:38:45.4023811Z 2022-08-17T12:38:45.4023876Z OK 2022-08-17T12:38:45.4024582Z 2022-08-17T12:38:45.4024685Z Generating XML reports... 2022-08-17T12:38:45.4058011Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123842.xml 2022-08-17T12:38:46.6961970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:46.6962370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:46.6964825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:46.6965210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:46.8470290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptja6z015 2022-08-17T12:38:46.8473835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptja6z015/_remote_module_non_scriptable.py 2022-08-17T12:38:47.1091493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:47.1102106Z 2022-08-17T12:38:47.1102211Z Running tests... 2022-08-17T12:38:47.1102673Z ---------------------------------------------------------------------- 2022-08-17T12:38:47.3742607Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6075 2022-08-17T12:38:47.3770441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6076 2022-08-17T12:38:47.3798933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6077 2022-08-17T12:38:47.3829771Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6078 2022-08-17T12:38:48.4337375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:48.4338030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:48.4340418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:48.4341082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:48.4729016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:48.4729697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:48.4733070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:48.4733701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:48.5015365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:48.5015997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:48.5018359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:48.5018982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:48.5038238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:48.5038962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:48.5042123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:48.5042866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:48.5902428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0nos7w5 2022-08-17T12:38:48.5903153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0nos7w5/_remote_module_non_scriptable.py 2022-08-17T12:38:48.6315587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo41f2t1e 2022-08-17T12:38:48.6316562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo41f2t1e/_remote_module_non_scriptable.py 2022-08-17T12:38:48.6593071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcspb04u 2022-08-17T12:38:48.6594080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcspb04u/_remote_module_non_scriptable.py 2022-08-17T12:38:48.6622301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_la75ito 2022-08-17T12:38:48.6623700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_la75ito/_remote_module_non_scriptable.py 2022-08-17T12:38:48.8554980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:48.8972318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:48.9255035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:48.9282839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:49.1997322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:38:49.2097654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:38:49.2100164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:49.2101698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:38:49.2102551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:38:49.2103767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:49.2105012Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:49.2106254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:49.2330232Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #5 to worker0, as the agent is shutting down 2022-08-17T12:38:49.2332366Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #4 to worker0, as the agent is shutting down 2022-08-17T12:38:49.2334236Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2337637Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2338233Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:49.2338791Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:49.2344211Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:49.2354829Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:49.2356425Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2357026Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2369788Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2371221Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-08-17T12:38:49.2372559Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #3 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-08-17T12:38:49.2373862Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2375520Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2386929Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2393879Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2395550Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2396612Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2413509Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #6 to worker1: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114) 2022-08-17T12:38:49.2414540Z [W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #7 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2415746Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:49.2416743Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:49.5889151Z ok (2.478s) 2022-08-17T12:38:49.5889394Z 2022-08-17T12:38:49.5889928Z ---------------------------------------------------------------------- 2022-08-17T12:38:49.5890413Z Ran 1 test in 2.479s 2022-08-17T12:38:49.5890627Z 2022-08-17T12:38:49.5890722Z OK 2022-08-17T12:38:49.5890898Z 2022-08-17T12:38:49.5891065Z Generating XML reports... 2022-08-17T12:38:49.5924070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123847.xml 2022-08-17T12:38:50.8976687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:50.8977325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:50.8979690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:50.8980297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:51.0499631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8l96shyt 2022-08-17T12:38:51.0501792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8l96shyt/_remote_module_non_scriptable.py 2022-08-17T12:38:51.3129395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:51.3140190Z 2022-08-17T12:38:51.3140367Z Running tests... 2022-08-17T12:38:51.3140779Z ---------------------------------------------------------------------- 2022-08-17T12:38:51.5801953Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6320 2022-08-17T12:38:51.5832830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6321 2022-08-17T12:38:51.5862015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6322 2022-08-17T12:38:51.5893886Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6323 2022-08-17T12:38:52.5775895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:52.5776551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:52.5779688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:52.6192707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:52.6193698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:52.6194306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:52.6196052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:52.6196621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:52.6419553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:52.6420244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:52.6424018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:52.6424721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:52.6550780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:52.6551509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:52.6554681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:52.6555377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:52.7365035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewta960o 2022-08-17T12:38:52.7366532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewta960o/_remote_module_non_scriptable.py 2022-08-17T12:38:52.7811430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yh6xfv1 2022-08-17T12:38:52.7812264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yh6xfv1/_remote_module_non_scriptable.py 2022-08-17T12:38:52.8023853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjh78q0i9 2022-08-17T12:38:52.8024556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjh78q0i9/_remote_module_non_scriptable.py 2022-08-17T12:38:52.8147328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuya3tw93 2022-08-17T12:38:52.8148108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuya3tw93/_remote_module_non_scriptable.py 2022-08-17T12:38:52.9990829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:53.0474281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:53.0649244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:53.0791398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:53.3103191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:38:53.3202139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:38:53.3304832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:38:53.3305831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:38:53.3307609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:53.3308879Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:53.3310372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:53.3311592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:38:53.3525050Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-08-17T12:38:53.3526907Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #6 to worker0, as the agent is shutting down 2022-08-17T12:38:53.3532025Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3533255Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3534514Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3535670Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3539054Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3540348Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3555016Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3582894Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #10 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3584648Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3585769Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3586934Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3588032Z [W tensorpipe_agent.cpp:679] RPC agent for worker1 encountered error when sending response to request #11 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3589282Z [W tensorpipe_agent.cpp:940] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3590645Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3591749Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3592760Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:38:53.3620238Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3621553Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.3622738Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:38:53.7952043Z ok (2.481s) 2022-08-17T12:38:53.7952293Z 2022-08-17T12:38:53.7952706Z ---------------------------------------------------------------------- 2022-08-17T12:38:53.7952965Z Ran 1 test in 2.481s 2022-08-17T12:38:53.7953088Z 2022-08-17T12:38:53.7953158Z OK 2022-08-17T12:38:53.7953253Z 2022-08-17T12:38:53.7953347Z Generating XML reports... 2022-08-17T12:38:53.7990347Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123851.xml 2022-08-17T12:38:55.1131132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:55.1131530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:55.1134337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:55.1134749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:55.2675317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c4i9wap 2022-08-17T12:38:55.2678121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c4i9wap/_remote_module_non_scriptable.py 2022-08-17T12:38:55.5306433Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:55.5317147Z 2022-08-17T12:38:55.5317291Z Running tests... 2022-08-17T12:38:55.5317929Z ---------------------------------------------------------------------- 2022-08-17T12:38:55.7957428Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6565 2022-08-17T12:38:55.7986301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6566 2022-08-17T12:38:55.8015098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6567 2022-08-17T12:38:55.8045319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6568 2022-08-17T12:38:56.8594502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:56.8595168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:56.8598494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:56.8599167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:56.9127623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:56.9128287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:56.9131019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:56.9131677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:56.9240325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:56.9241006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:56.9243711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:56.9244293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:56.9246876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:56.9247741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:56.9251940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:56.9252572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:57.0175473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8uuylf_ 2022-08-17T12:38:57.0176865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8uuylf_/_remote_module_non_scriptable.py 2022-08-17T12:38:57.0704537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmlo8ug4 2022-08-17T12:38:57.0705407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmlo8ug4/_remote_module_non_scriptable.py 2022-08-17T12:38:57.0822850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwblc2l5 2022-08-17T12:38:57.0823743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwblc2l5/_remote_module_non_scriptable.py 2022-08-17T12:38:57.0834921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn85gp0hl 2022-08-17T12:38:57.0837127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn85gp0hl/_remote_module_non_scriptable.py 2022-08-17T12:38:57.2779210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:38:57.3302427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:38:57.3425638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:38:57.3448463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:38:58.0107293Z ok (2.479s) 2022-08-17T12:38:58.0107536Z 2022-08-17T12:38:58.0108087Z ---------------------------------------------------------------------- 2022-08-17T12:38:58.0108439Z Ran 1 test in 2.479s 2022-08-17T12:38:58.0108562Z 2022-08-17T12:38:58.0108612Z OK 2022-08-17T12:38:58.0108708Z 2022-08-17T12:38:58.0108805Z Generating XML reports... 2022-08-17T12:38:58.0142870Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123855.xml 2022-08-17T12:38:59.3063808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:38:59.3064243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:38:59.3067310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:38:59.3067686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:38:59.4553442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnk3ox8h 2022-08-17T12:38:59.4556694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnk3ox8h/_remote_module_non_scriptable.py 2022-08-17T12:38:59.7177607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:38:59.7188273Z 2022-08-17T12:38:59.7188406Z Running tests... 2022-08-17T12:38:59.7188771Z ---------------------------------------------------------------------- 2022-08-17T12:38:59.9826558Z test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6808 2022-08-17T12:38:59.9855363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6809 2022-08-17T12:38:59.9885565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6810 2022-08-17T12:38:59.9924305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6811 2022-08-17T12:39:01.0121055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:01.0122021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:01.0125162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:01.0126055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:01.0402270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:01.0402880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:01.0407241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:01.0407902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:01.0408668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:01.0409245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:01.0411835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:01.0412473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:01.0478470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:01.0479110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:01.0482745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:01.0483412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:01.1708933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmpp21mv 2022-08-17T12:39:01.1709784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmpp21mv/_remote_module_non_scriptable.py 2022-08-17T12:39:01.1977875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw6q8jej 2022-08-17T12:39:01.1978607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw6q8jej/_remote_module_non_scriptable.py 2022-08-17T12:39:01.1985413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxf09bqb 2022-08-17T12:39:01.1987750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxf09bqb/_remote_module_non_scriptable.py 2022-08-17T12:39:01.2049990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu20s39te 2022-08-17T12:39:01.2052429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu20s39te/_remote_module_non_scriptable.py 2022-08-17T12:39:01.4299357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:01.4552426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:01.4586313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:01.4648801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:02.0983294Z ok (2.379s) 2022-08-17T12:39:02.0983508Z 2022-08-17T12:39:02.0984067Z ---------------------------------------------------------------------- 2022-08-17T12:39:02.0984414Z Ran 1 test in 2.379s 2022-08-17T12:39:02.0984535Z 2022-08-17T12:39:02.0984597Z OK 2022-08-17T12:39:02.0984691Z 2022-08-17T12:39:02.0984788Z Generating XML reports... 2022-08-17T12:39:02.1017684Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123859.xml 2022-08-17T12:39:03.4122468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:03.4122869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:03.4125666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:03.4126204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:03.5618313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3xv941q 2022-08-17T12:39:03.5621227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3xv941q/_remote_module_non_scriptable.py 2022-08-17T12:39:03.8259583Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:03.8269484Z 2022-08-17T12:39:03.8269635Z Running tests... 2022-08-17T12:39:03.8269999Z ---------------------------------------------------------------------- 2022-08-17T12:39:04.0945031Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7051 2022-08-17T12:39:04.0973020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7052 2022-08-17T12:39:04.1003498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7053 2022-08-17T12:39:04.1032825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7054 2022-08-17T12:39:05.1085138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:05.1086042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:05.1088603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:05.1089283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:05.1457537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:05.1458227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:05.1460223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:05.1460895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:05.1527782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:05.1528480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:05.1531770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:05.1532378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:05.1632542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:05.1633220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:05.1637103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:05.1637762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:05.2664025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0ulmhdg 2022-08-17T12:39:05.2664919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0ulmhdg/_remote_module_non_scriptable.py 2022-08-17T12:39:05.3060273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue4l98vy 2022-08-17T12:39:05.3061013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue4l98vy/_remote_module_non_scriptable.py 2022-08-17T12:39:05.3094822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjpqrqkr 2022-08-17T12:39:05.3096586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjpqrqkr/_remote_module_non_scriptable.py 2022-08-17T12:39:05.3216306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuidv_0bh 2022-08-17T12:39:05.3217618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuidv_0bh/_remote_module_non_scriptable.py 2022-08-17T12:39:05.5262285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:05.5674570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:05.5675217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:05.5854268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:06.2092432Z ok (2.382s) 2022-08-17T12:39:06.2092666Z 2022-08-17T12:39:06.2093205Z ---------------------------------------------------------------------- 2022-08-17T12:39:06.2093502Z Ran 1 test in 2.382s 2022-08-17T12:39:06.2093626Z 2022-08-17T12:39:06.2093690Z OK 2022-08-17T12:39:06.2093786Z 2022-08-17T12:39:06.2093880Z Generating XML reports... 2022-08-17T12:39:06.2126282Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123903.xml 2022-08-17T12:39:07.5168213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:07.5169230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:07.5172435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:07.5173047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:07.6694494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwb1ogwf 2022-08-17T12:39:07.6699019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwb1ogwf/_remote_module_non_scriptable.py 2022-08-17T12:39:07.9337775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:07.9349088Z 2022-08-17T12:39:07.9349365Z Running tests... 2022-08-17T12:39:07.9350050Z ---------------------------------------------------------------------- 2022-08-17T12:39:08.1974363Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7294 2022-08-17T12:39:08.2002698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7295 2022-08-17T12:39:08.2035139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7296 2022-08-17T12:39:08.2067264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7297 2022-08-17T12:39:09.2098819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:09.2099425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:09.2103803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:09.2104535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:09.2142456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:09.2143066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:09.2147572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:09.2148241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:09.2445318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:09.2446152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:09.2448857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:09.2449807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:09.2710121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:09.2710667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:09.2713529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:09.2714538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:09.3729747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_0gvm1b 2022-08-17T12:39:09.3730843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhystha1 2022-08-17T12:39:09.3731824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_0gvm1b/_remote_module_non_scriptable.py 2022-08-17T12:39:09.3732875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhystha1/_remote_module_non_scriptable.py 2022-08-17T12:39:09.4087592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlqtn4ef 2022-08-17T12:39:09.4089323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlqtn4ef/_remote_module_non_scriptable.py 2022-08-17T12:39:09.4363582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92d0c796 2022-08-17T12:39:09.4364303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92d0c796/_remote_module_non_scriptable.py 2022-08-17T12:39:09.6349564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:09.6377694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:09.6728044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:09.7001732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:10.4127551Z ok (2.477s) 2022-08-17T12:39:10.4127821Z 2022-08-17T12:39:10.4128210Z ---------------------------------------------------------------------- 2022-08-17T12:39:10.4128529Z Ran 1 test in 2.478s 2022-08-17T12:39:10.4128657Z 2022-08-17T12:39:10.4129852Z OK 2022-08-17T12:39:10.4130003Z 2022-08-17T12:39:10.4130162Z Generating XML reports... 2022-08-17T12:39:10.4162089Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123907.xml 2022-08-17T12:39:11.7135898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:11.7136289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:11.7138452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:11.7138814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:11.8626791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2cw_sz1 2022-08-17T12:39:11.8630447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2cw_sz1/_remote_module_non_scriptable.py 2022-08-17T12:39:12.1272585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:12.1283800Z 2022-08-17T12:39:12.1284242Z Running tests... 2022-08-17T12:39:12.1284889Z ---------------------------------------------------------------------- 2022-08-17T12:39:12.3910937Z test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7537 2022-08-17T12:39:12.3939604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7538 2022-08-17T12:39:12.3967356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7539 2022-08-17T12:39:12.4000704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7540 2022-08-17T12:39:13.3885474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:13.3886364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:13.3888813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:13.3889496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:13.4380359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:13.4381060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:13.4384606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:13.4385573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:13.4431861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:13.4432490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:13.4438031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:13.4438682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:13.4487179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:13.4487807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:13.4492754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:13.4493370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:13.5495379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tlg13rp 2022-08-17T12:39:13.5496320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tlg13rp/_remote_module_non_scriptable.py 2022-08-17T12:39:13.5964433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq23ttnc5 2022-08-17T12:39:13.5965249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq23ttnc5/_remote_module_non_scriptable.py 2022-08-17T12:39:13.6128194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t6rgqjm 2022-08-17T12:39:13.6129008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t6rgqjm/_remote_module_non_scriptable.py 2022-08-17T12:39:13.6168417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7rflt_m 2022-08-17T12:39:13.6169943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7rflt_m/_remote_module_non_scriptable.py 2022-08-17T12:39:13.8071757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:13.8554607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:13.8747701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:13.8765537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:14.6060954Z ok (2.477s) 2022-08-17T12:39:14.6061201Z 2022-08-17T12:39:14.6061757Z ---------------------------------------------------------------------- 2022-08-17T12:39:14.6062043Z Ran 1 test in 2.478s 2022-08-17T12:39:14.6062163Z 2022-08-17T12:39:14.6062226Z OK 2022-08-17T12:39:14.6062321Z 2022-08-17T12:39:14.6062412Z Generating XML reports... 2022-08-17T12:39:14.6094479Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123912.xml 2022-08-17T12:39:15.9058615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:15.9059282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:15.9061484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:15.9062042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:16.0589540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1dnf44m 2022-08-17T12:39:16.0592535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1dnf44m/_remote_module_non_scriptable.py 2022-08-17T12:39:16.3244225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:16.3254968Z 2022-08-17T12:39:16.3255277Z Running tests... 2022-08-17T12:39:16.3255901Z ---------------------------------------------------------------------- 2022-08-17T12:39:16.5892565Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7780 2022-08-17T12:39:16.5926313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7781 2022-08-17T12:39:16.5961719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7782 2022-08-17T12:39:16.5992429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7783 2022-08-17T12:39:17.6290903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:17.6291644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:17.6294971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:17.6295598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:17.6752007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:17.6752693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:17.6755306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:17.6755951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:17.6907234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:17.6907783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:17.6911019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:17.6911582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:17.6994085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:17.6997192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:17.6998065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:17.6998691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:17.7898085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps74a7p5l 2022-08-17T12:39:17.7899149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps74a7p5l/_remote_module_non_scriptable.py 2022-08-17T12:39:17.8347303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuje09qt 2022-08-17T12:39:17.8348229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuje09qt/_remote_module_non_scriptable.py 2022-08-17T12:39:17.8535987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3w4sv2z 2022-08-17T12:39:17.8537402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3w4sv2z/_remote_module_non_scriptable.py 2022-08-17T12:39:17.8589859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1zykes8 2022-08-17T12:39:17.8591817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1zykes8/_remote_module_non_scriptable.py 2022-08-17T12:39:18.0503399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:18.0921597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:18.1159484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:18.1204172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:18.7049584Z ok (2.379s) 2022-08-17T12:39:18.7049811Z 2022-08-17T12:39:18.7050379Z ---------------------------------------------------------------------- 2022-08-17T12:39:18.7050744Z Ran 1 test in 2.379s 2022-08-17T12:39:18.7050871Z 2022-08-17T12:39:18.7050938Z OK 2022-08-17T12:39:18.7051032Z 2022-08-17T12:39:18.7051133Z Generating XML reports... 2022-08-17T12:39:18.7086512Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123916.xml 2022-08-17T12:39:20.0022514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:20.0022931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:20.0025425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:20.0025787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:20.1531331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfs_t0k4 2022-08-17T12:39:20.1534875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfs_t0k4/_remote_module_non_scriptable.py 2022-08-17T12:39:20.4147468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:20.4157353Z 2022-08-17T12:39:20.4157499Z Running tests... 2022-08-17T12:39:20.4158657Z ---------------------------------------------------------------------- 2022-08-17T12:39:20.6729051Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8023 2022-08-17T12:39:20.6757970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8024 2022-08-17T12:39:20.6787180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8025 2022-08-17T12:39:20.6817488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8026 2022-08-17T12:39:21.7165584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:21.7166442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:21.7168525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:21.7169098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:21.7174817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:21.7175474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:21.7179277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:21.7179923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:21.7235527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:21.7236188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:21.7240197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:21.7240824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:21.7352969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:21.7353651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:21.7356286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:21.7356922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:21.8795319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6nbg3gm 2022-08-17T12:39:21.8796363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6nbg3gm/_remote_module_non_scriptable.py 2022-08-17T12:39:21.8881925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm26ce9zi 2022-08-17T12:39:21.8883420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm26ce9zi/_remote_module_non_scriptable.py 2022-08-17T12:39:21.8936270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eti0z_1 2022-08-17T12:39:21.8937741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eti0z_1/_remote_module_non_scriptable.py 2022-08-17T12:39:21.8961768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6_yp9mj 2022-08-17T12:39:21.8964075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6_yp9mj/_remote_module_non_scriptable.py 2022-08-17T12:39:22.1386895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:22.1500673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:22.1518036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:22.1580451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:22.9880385Z ok (2.572s) 2022-08-17T12:39:22.9880661Z 2022-08-17T12:39:22.9881157Z ---------------------------------------------------------------------- 2022-08-17T12:39:22.9881572Z Ran 1 test in 2.572s 2022-08-17T12:39:22.9881765Z 2022-08-17T12:39:22.9881873Z OK 2022-08-17T12:39:22.9882014Z 2022-08-17T12:39:22.9882153Z Generating XML reports... 2022-08-17T12:39:22.9917108Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123920.xml 2022-08-17T12:39:24.2733767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:24.2734153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:24.2736532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:24.2736929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:24.4245449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp116zs7a4 2022-08-17T12:39:24.4248490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp116zs7a4/_remote_module_non_scriptable.py 2022-08-17T12:39:24.6860265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:24.6871247Z 2022-08-17T12:39:24.6871574Z Running tests... 2022-08-17T12:39:24.6871973Z ---------------------------------------------------------------------- 2022-08-17T12:39:24.9525556Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8266 2022-08-17T12:39:24.9552343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8267 2022-08-17T12:39:24.9582503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8268 2022-08-17T12:39:24.9611507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8269 2022-08-17T12:39:25.9573359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:25.9574062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:25.9577735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:25.9578362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:25.9850742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:25.9851770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:25.9854274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:25.9854982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:26.0237507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:26.0238465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:26.0253431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:26.0254138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:26.0531633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:26.0532286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:26.0535834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:26.0536514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:26.1191445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5nd6wdl 2022-08-17T12:39:26.1192305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5nd6wdl/_remote_module_non_scriptable.py 2022-08-17T12:39:26.1441795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec2qbzr2 2022-08-17T12:39:26.1443426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec2qbzr2/_remote_module_non_scriptable.py 2022-08-17T12:39:26.1830388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcht4p952 2022-08-17T12:39:26.1831357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcht4p952/_remote_module_non_scriptable.py 2022-08-17T12:39:26.2122829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lqjr4cn 2022-08-17T12:39:26.2123573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lqjr4cn/_remote_module_non_scriptable.py 2022-08-17T12:39:26.3833870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:26.4056990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:26.4430689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:26.4752642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:27.1671987Z ok (2.480s) 2022-08-17T12:39:27.1672275Z 2022-08-17T12:39:27.1672826Z ---------------------------------------------------------------------- 2022-08-17T12:39:27.1673120Z Ran 1 test in 2.480s 2022-08-17T12:39:27.1673244Z 2022-08-17T12:39:27.1673294Z OK 2022-08-17T12:39:27.1673389Z 2022-08-17T12:39:27.1673486Z Generating XML reports... 2022-08-17T12:39:27.1706917Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123924.xml 2022-08-17T12:39:28.4734269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:28.4734678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:28.4736551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:28.4736955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:28.6242275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjz8vmfp 2022-08-17T12:39:28.6245358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjz8vmfp/_remote_module_non_scriptable.py 2022-08-17T12:39:28.8862165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:28.8872433Z 2022-08-17T12:39:28.8872594Z Running tests... 2022-08-17T12:39:28.8873323Z ---------------------------------------------------------------------- 2022-08-17T12:39:29.1545967Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8509 2022-08-17T12:39:29.1574162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8510 2022-08-17T12:39:29.1602097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8511 2022-08-17T12:39:29.1634105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8512 2022-08-17T12:39:30.1400075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:30.1400775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:30.1403528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:30.1404191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:30.2067858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:30.2068502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:30.2071400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:30.2072025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:30.2106441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:30.2107189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:30.2110462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:30.2111132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:30.2170253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:30.2170893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:30.2174312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:30.2174979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:30.3020695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yuvunph 2022-08-17T12:39:30.3022051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yuvunph/_remote_module_non_scriptable.py 2022-08-17T12:39:30.3676486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvssog56 2022-08-17T12:39:30.3677292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvssog56/_remote_module_non_scriptable.py 2022-08-17T12:39:30.3727870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahqdxfus 2022-08-17T12:39:30.3729260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahqdxfus/_remote_module_non_scriptable.py 2022-08-17T12:39:30.3770763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcor0v2_ 2022-08-17T12:39:30.3772685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcor0v2_/_remote_module_non_scriptable.py 2022-08-17T12:39:30.5669968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:30.6314631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:30.6404511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:30.6414805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:41.2843920Z ok (12.397s) 2022-08-17T12:39:41.2844169Z 2022-08-17T12:39:41.2844704Z ---------------------------------------------------------------------- 2022-08-17T12:39:41.2844988Z Ran 1 test in 12.397s 2022-08-17T12:39:41.2845111Z 2022-08-17T12:39:41.2845162Z OK 2022-08-17T12:39:41.2845257Z 2022-08-17T12:39:41.2845351Z Generating XML reports... 2022-08-17T12:39:41.2879008Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123928.xml 2022-08-17T12:39:42.5810497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:42.5810909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:42.5813106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:42.5813561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:42.7334172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcknzymx 2022-08-17T12:39:42.7337059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcknzymx/_remote_module_non_scriptable.py 2022-08-17T12:39:42.9984975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:42.9995753Z 2022-08-17T12:39:42.9995902Z Running tests... 2022-08-17T12:39:42.9996550Z ---------------------------------------------------------------------- 2022-08-17T12:39:43.2569915Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s) 2022-08-17T12:39:43.2570449Z 2022-08-17T12:39:43.2570663Z ---------------------------------------------------------------------- 2022-08-17T12:39:43.2570931Z Ran 1 test in 0.257s 2022-08-17T12:39:43.2571050Z 2022-08-17T12:39:43.2571127Z OK (skipped=1) 2022-08-17T12:39:43.2571240Z 2022-08-17T12:39:43.2571329Z Generating XML reports... 2022-08-17T12:39:43.2595005Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123942.xml 2022-08-17T12:39:44.5072094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:44.5072483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:44.5075122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:44.5075484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:44.6590558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c3v1ogk 2022-08-17T12:39:44.6593930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c3v1ogk/_remote_module_non_scriptable.py 2022-08-17T12:39:44.9194506Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:44.9204827Z 2022-08-17T12:39:44.9204928Z Running tests... 2022-08-17T12:39:44.9205428Z ---------------------------------------------------------------------- 2022-08-17T12:39:45.1795218Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8766 2022-08-17T12:39:45.1825147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8767 2022-08-17T12:39:45.1855127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8768 2022-08-17T12:39:45.1887100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8769 2022-08-17T12:39:46.1776851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:46.1777534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:46.1779146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:46.1780071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:46.2138466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:46.2139240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:46.2142227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:46.2143433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:46.2240758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:46.2241443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:46.2245690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:46.2246559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:46.2445173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:46.2446007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:46.2448720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:46.2449485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:46.3552199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbveg5j6j 2022-08-17T12:39:46.3553044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbveg5j6j/_remote_module_non_scriptable.py 2022-08-17T12:39:46.3751629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2cjdt4e 2022-08-17T12:39:46.3752490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2cjdt4e/_remote_module_non_scriptable.py 2022-08-17T12:39:46.3861380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafvko7w_ 2022-08-17T12:39:46.3864520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafvko7w_/_remote_module_non_scriptable.py 2022-08-17T12:39:46.4086836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb2zrchg 2022-08-17T12:39:46.4087698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb2zrchg/_remote_module_non_scriptable.py 2022-08-17T12:39:46.6158925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:46.6356754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:46.6459443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:46.6694257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:47.3944407Z ok (2.474s) 2022-08-17T12:39:47.3944675Z 2022-08-17T12:39:47.3945175Z ---------------------------------------------------------------------- 2022-08-17T12:39:47.3945466Z Ran 1 test in 2.474s 2022-08-17T12:39:47.3945592Z 2022-08-17T12:39:47.3945658Z OK 2022-08-17T12:39:47.3945739Z 2022-08-17T12:39:47.3945835Z Generating XML reports... 2022-08-17T12:39:47.3978946Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123944.xml 2022-08-17T12:39:48.6784149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:48.6784552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:48.6786967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:48.6787460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:48.8311694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmepnrph0 2022-08-17T12:39:48.8314667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmepnrph0/_remote_module_non_scriptable.py 2022-08-17T12:39:49.0961952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:49.0972950Z 2022-08-17T12:39:49.0973066Z Running tests... 2022-08-17T12:39:49.0973686Z ---------------------------------------------------------------------- 2022-08-17T12:39:49.3596863Z test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9009 2022-08-17T12:39:49.3627261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9010 2022-08-17T12:39:49.3656499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9011 2022-08-17T12:39:49.3687783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9012 2022-08-17T12:39:50.3440759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:50.3441448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:50.3444737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:50.3445417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:50.3550658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:50.3551312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:50.3555410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:50.3556016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:50.4022021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:50.4022721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:50.4027390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:50.4027964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:50.4136058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:50.4136604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:50.4140402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:50.4141063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:50.5077762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9j5d3ue 2022-08-17T12:39:50.5078772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9j5d3ue/_remote_module_non_scriptable.py 2022-08-17T12:39:50.5197856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7dd2cqa 2022-08-17T12:39:50.5198714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7dd2cqa/_remote_module_non_scriptable.py 2022-08-17T12:39:50.5715182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wge1hs7 2022-08-17T12:39:50.5716488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wge1hs7/_remote_module_non_scriptable.py 2022-08-17T12:39:50.5789084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph112xffc 2022-08-17T12:39:50.5790151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph112xffc/_remote_module_non_scriptable.py 2022-08-17T12:39:50.7637136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:50.7786360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:50.8320389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:50.8392434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:51.4746829Z ok (2.377s) 2022-08-17T12:39:51.4747101Z 2022-08-17T12:39:51.4747586Z ---------------------------------------------------------------------- 2022-08-17T12:39:51.4748008Z Ran 1 test in 2.377s 2022-08-17T12:39:51.4748221Z 2022-08-17T12:39:51.4748331Z OK 2022-08-17T12:39:51.4748509Z 2022-08-17T12:39:51.4748691Z Generating XML reports... 2022-08-17T12:39:51.4782324Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123949.xml 2022-08-17T12:39:52.7761749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:52.7762155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:52.7764582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:52.7764961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:52.9291685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8w7zyjb 2022-08-17T12:39:52.9294889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8w7zyjb/_remote_module_non_scriptable.py 2022-08-17T12:39:53.1928037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:53.1938649Z 2022-08-17T12:39:53.1938833Z Running tests... 2022-08-17T12:39:53.1939274Z ---------------------------------------------------------------------- 2022-08-17T12:39:53.4573967Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9236 2022-08-17T12:39:53.4602353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9237 2022-08-17T12:39:53.4630947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9238 2022-08-17T12:39:53.4662329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9239 2022-08-17T12:39:54.4928524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:54.4929155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:54.4932098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:54.4933034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:54.5180440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:54.5181092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:54.5185142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:54.5185836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:54.5243094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:54.5243771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:54.5248304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:54.5248991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:54.5261918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:54.5262824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:54.5266310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:54.5267031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:54.6517908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yfpotau 2022-08-17T12:39:54.6518741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yfpotau/_remote_module_non_scriptable.py 2022-08-17T12:39:54.6824769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_nxnyvf 2022-08-17T12:39:54.6825600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_nxnyvf/_remote_module_non_scriptable.py 2022-08-17T12:39:54.6935956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s6i_4_f 2022-08-17T12:39:54.6936940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s6i_4_f/_remote_module_non_scriptable.py 2022-08-17T12:39:54.7028271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzvyqcvd 2022-08-17T12:39:54.7029136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzvyqcvd/_remote_module_non_scriptable.py 2022-08-17T12:39:54.9121896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:54.9466696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:54.9563082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:54.9648735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:55.5720863Z ok (2.378s) 2022-08-17T12:39:55.5721077Z 2022-08-17T12:39:55.5721416Z ---------------------------------------------------------------------- 2022-08-17T12:39:55.5721669Z Ran 1 test in 2.378s 2022-08-17T12:39:55.5721818Z 2022-08-17T12:39:55.5722388Z OK 2022-08-17T12:39:55.5722542Z 2022-08-17T12:39:55.5722711Z Generating XML reports... 2022-08-17T12:39:55.5756546Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123953.xml 2022-08-17T12:39:56.8918589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:56.8919018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:56.8921017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:56.8921532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:57.0418603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi9bk8fu 2022-08-17T12:39:57.0422132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi9bk8fu/_remote_module_non_scriptable.py 2022-08-17T12:39:57.3051621Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:39:57.3062202Z 2022-08-17T12:39:57.3062306Z Running tests... 2022-08-17T12:39:57.3062908Z ---------------------------------------------------------------------- 2022-08-17T12:39:57.5728483Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9479 2022-08-17T12:39:57.5758379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9480 2022-08-17T12:39:57.5787251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9481 2022-08-17T12:39:57.5817637Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9482 2022-08-17T12:39:58.5901772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:58.5902441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:58.5905660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:58.5906322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:58.6207087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:58.6207731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:58.6209541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:58.6210148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:58.6239083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:58.6239753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:58.6243805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:58.6244418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:58.6285670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:39:58.6286510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:39:58.6290771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:39:58.6291358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:39:58.7562444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1jhehu2 2022-08-17T12:39:58.7563877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1jhehu2/_remote_module_non_scriptable.py 2022-08-17T12:39:58.7809965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm52_oeb 2022-08-17T12:39:58.7812351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm52_oeb/_remote_module_non_scriptable.py 2022-08-17T12:39:58.7849185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e50rb9y 2022-08-17T12:39:58.7851037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e50rb9y/_remote_module_non_scriptable.py 2022-08-17T12:39:58.7927054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4otj_dhr 2022-08-17T12:39:58.7929562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4otj_dhr/_remote_module_non_scriptable.py 2022-08-17T12:39:59.0303942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:39:59.0455061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:39:59.0527709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:39:59.0568767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:39:59.7879227Z ok (2.481s) 2022-08-17T12:39:59.7879505Z 2022-08-17T12:39:59.7880059Z ---------------------------------------------------------------------- 2022-08-17T12:39:59.7880391Z Ran 1 test in 2.482s 2022-08-17T12:39:59.7880515Z 2022-08-17T12:39:59.7880569Z OK 2022-08-17T12:39:59.7880670Z 2022-08-17T12:39:59.7880769Z Generating XML reports... 2022-08-17T12:39:59.7913857Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123957.xml 2022-08-17T12:40:01.0862701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:01.0863404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:01.0866564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:01.0867227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:01.2395684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0083tvho 2022-08-17T12:40:01.2399353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0083tvho/_remote_module_non_scriptable.py 2022-08-17T12:40:01.5080947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:01.5091398Z 2022-08-17T12:40:01.5091547Z Running tests... 2022-08-17T12:40:01.5092169Z ---------------------------------------------------------------------- 2022-08-17T12:40:01.5105816Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:40:01.7750384Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9722 2022-08-17T12:40:01.7778115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9723 2022-08-17T12:40:01.7806865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9724 2022-08-17T12:40:01.7839367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9725 2022-08-17T12:40:02.8473237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:02.8473868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:02.8476294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:02.8476947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:02.8631186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:02.8631808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:02.8633962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:02.8634608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:02.8666693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:02.8667284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:02.8671701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:02.8672340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:02.8848479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:02.8849090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:02.8852111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:02.8852766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:03.0225006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo9h4f_t 2022-08-17T12:40:03.0225970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo9h4f_t/_remote_module_non_scriptable.py 2022-08-17T12:40:03.0298819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2drc3f_ 2022-08-17T12:40:03.0299774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph23uoazu 2022-08-17T12:40:03.0300572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2drc3f_/_remote_module_non_scriptable.py 2022-08-17T12:40:03.0301340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph23uoazu/_remote_module_non_scriptable.py 2022-08-17T12:40:03.0433335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rt_7dkk 2022-08-17T12:40:03.0434091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rt_7dkk/_remote_module_non_scriptable.py 2022-08-17T12:40:03.2861414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:03.2932765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:03.2953316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:03.3046009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:03.5378129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:03.5478077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:03.5586564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:03.5587868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:03.5589000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:03.5589998Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:03.5590729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:03.5591521Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:03.7337189Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7342690Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7347554Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7427134Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7428128Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7429478Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7449857Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7450604Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7451167Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7569430Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7570461Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:03.7571657Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:40:04.1901764Z ok (2.681s) 2022-08-17T12:40:04.1902029Z 2022-08-17T12:40:04.1902559Z ---------------------------------------------------------------------- 2022-08-17T12:40:04.1902821Z Ran 1 test in 2.681s 2022-08-17T12:40:04.1902930Z 2022-08-17T12:40:04.1902995Z OK 2022-08-17T12:40:04.1903092Z 2022-08-17T12:40:04.1903186Z Generating XML reports... 2022-08-17T12:40:04.1937942Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124001.xml 2022-08-17T12:40:05.5157316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:05.5157906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:05.5168013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:05.5168735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:05.6694521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzgrowem 2022-08-17T12:40:05.6696965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzgrowem/_remote_module_non_scriptable.py 2022-08-17T12:40:05.9403665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:05.9414699Z 2022-08-17T12:40:05.9415062Z Running tests... 2022-08-17T12:40:05.9415502Z ---------------------------------------------------------------------- 2022-08-17T12:40:06.2095386Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9961 2022-08-17T12:40:06.2123687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9962 2022-08-17T12:40:06.2153682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9963 2022-08-17T12:40:06.2188042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9964 2022-08-17T12:40:07.2875677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:07.2876305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:07.2878718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:07.2879663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:07.2928816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:07.2929484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:07.2933307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:07.2933919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:07.3293662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:07.3294312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:07.3296835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:07.3297475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:07.3566393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:07.3566924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:07.3570379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:07.3570913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:07.4488195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxesreox3 2022-08-17T12:40:07.4489019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxesreox3/_remote_module_non_scriptable.py 2022-08-17T12:40:07.4557918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez6pr4nf 2022-08-17T12:40:07.4558635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez6pr4nf/_remote_module_non_scriptable.py 2022-08-17T12:40:07.4899466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja1fdmim 2022-08-17T12:40:07.4900513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja1fdmim/_remote_module_non_scriptable.py 2022-08-17T12:40:07.5157098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07_8malh 2022-08-17T12:40:07.5158180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07_8malh/_remote_module_non_scriptable.py 2022-08-17T12:40:07.7161474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:07.7208163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:07.7546465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:07.7796610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:08.0322824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:08.0324608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:08.0326508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:08.0327391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:08.0328723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:08.0431044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:08.0431946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:08.0432828Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:08.6253092Z ok (2.683s) 2022-08-17T12:40:08.6253388Z 2022-08-17T12:40:08.6253882Z ---------------------------------------------------------------------- 2022-08-17T12:40:08.6254149Z Ran 1 test in 2.684s 2022-08-17T12:40:08.6254280Z 2022-08-17T12:40:08.6254347Z OK 2022-08-17T12:40:08.6254443Z 2022-08-17T12:40:08.6254539Z Generating XML reports... 2022-08-17T12:40:08.6288362Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124005.xml 2022-08-17T12:40:09.9421683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:09.9422364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:09.9425685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:09.9426393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:10.0949464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_clvlelg 2022-08-17T12:40:10.0953417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_clvlelg/_remote_module_non_scriptable.py 2022-08-17T12:40:10.3619822Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:10.3631099Z 2022-08-17T12:40:10.3631435Z Running tests... 2022-08-17T12:40:10.3632135Z ---------------------------------------------------------------------- 2022-08-17T12:40:10.6295393Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10196 2022-08-17T12:40:10.6324972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10197 2022-08-17T12:40:10.6355075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10198 2022-08-17T12:40:10.6386507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10199 2022-08-17T12:40:11.6918538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:11.6919164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:11.6922193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:11.6922835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:11.7066727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:11.7067282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:11.7070629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:11.7071213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:11.7551271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:11.7551834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:11.7555050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:11.7555617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:11.7556295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:11.7556785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:11.7561277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:11.7561836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:11.8511220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjethg_b 2022-08-17T12:40:11.8512568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjethg_b/_remote_module_non_scriptable.py 2022-08-17T12:40:11.8651909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3jg27b_ 2022-08-17T12:40:11.8652695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3jg27b_/_remote_module_non_scriptable.py 2022-08-17T12:40:11.9136605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25bzlyj0 2022-08-17T12:40:11.9137456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25bzlyj0/_remote_module_non_scriptable.py 2022-08-17T12:40:11.9146774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4eksmdg 2022-08-17T12:40:11.9148640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4eksmdg/_remote_module_non_scriptable.py 2022-08-17T12:40:12.1122220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:12.1237861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:12.1714939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:12.1724466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:12.4593512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:12.4695987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:12.4798193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:12.4798958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:12.4799409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:12.4802057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:12.4802767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:12.4897864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:13.0450960Z ok (2.682s) 2022-08-17T12:40:13.0451215Z 2022-08-17T12:40:13.0451733Z ---------------------------------------------------------------------- 2022-08-17T12:40:13.0452038Z Ran 1 test in 2.682s 2022-08-17T12:40:13.0452162Z 2022-08-17T12:40:13.0452227Z OK 2022-08-17T12:40:13.0452324Z 2022-08-17T12:40:13.0452423Z Generating XML reports... 2022-08-17T12:40:13.0485831Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124010.xml 2022-08-17T12:40:14.3571349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:14.3571754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:14.3574592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:14.3574955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:14.5113700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeizvea8n 2022-08-17T12:40:14.5117712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeizvea8n/_remote_module_non_scriptable.py 2022-08-17T12:40:14.7752719Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:14.7764349Z 2022-08-17T12:40:14.7764672Z Running tests... 2022-08-17T12:40:14.7765671Z ---------------------------------------------------------------------- 2022-08-17T12:40:15.0401205Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10431 2022-08-17T12:40:15.0430298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10432 2022-08-17T12:40:15.0459211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10433 2022-08-17T12:40:15.0492417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10434 2022-08-17T12:40:16.0625676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:16.0626426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:16.0629298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:16.0629979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:16.0995158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:16.0995820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:16.0999097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:16.0999652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:16.1742820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:16.1743486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:16.1747113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:16.1747780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:16.1783288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:16.1783890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:16.1788091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:16.1788751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:16.2203175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvm52xia 2022-08-17T12:40:16.2203880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvm52xia/_remote_module_non_scriptable.py 2022-08-17T12:40:16.2579860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gxwxqvt 2022-08-17T12:40:16.2580647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gxwxqvt/_remote_module_non_scriptable.py 2022-08-17T12:40:16.3334996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju37szpd 2022-08-17T12:40:16.3335735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju37szpd/_remote_module_non_scriptable.py 2022-08-17T12:40:16.3374995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnypxcgv 2022-08-17T12:40:16.3376254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnypxcgv/_remote_module_non_scriptable.py 2022-08-17T12:40:16.4820983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:16.5214309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:16.5985543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:16.6023438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:16.8440809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:16.8541283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:16.8643157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:16.8646105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:16.8647544Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:16.8648124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:16.8648757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:16.8649368Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:17.4554484Z ok (2.679s) 2022-08-17T12:40:17.4555215Z 2022-08-17T12:40:17.4555696Z ---------------------------------------------------------------------- 2022-08-17T12:40:17.4556189Z Ran 1 test in 2.679s 2022-08-17T12:40:17.4556406Z 2022-08-17T12:40:17.4556524Z OK 2022-08-17T12:40:17.4557018Z 2022-08-17T12:40:17.4557170Z Generating XML reports... 2022-08-17T12:40:17.4590204Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124014.xml 2022-08-17T12:40:18.7705589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:18.7706351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:18.7709131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:18.7709708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:18.9237852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr6mpp49 2022-08-17T12:40:18.9241783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr6mpp49/_remote_module_non_scriptable.py 2022-08-17T12:40:19.1885212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:19.1896990Z 2022-08-17T12:40:19.1897296Z Running tests... 2022-08-17T12:40:19.1897969Z ---------------------------------------------------------------------- 2022-08-17T12:40:19.4537808Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10666 2022-08-17T12:40:19.4567181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10667 2022-08-17T12:40:19.4595298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10668 2022-08-17T12:40:19.4625905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10669 2022-08-17T12:40:20.5368647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:20.5369286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:20.5370970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:20.5371600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:20.5538338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:20.5538999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:20.5542387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:20.5543308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:20.5575203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:20.5575835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:20.5579962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:20.5580634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:20.6086656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:20.6087242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:20.6089733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:20.6090283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:20.7183327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9sfgay7 2022-08-17T12:40:20.7184490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9sfgay7/_remote_module_non_scriptable.py 2022-08-17T12:40:20.7192818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxujigqod 2022-08-17T12:40:20.7194797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxujigqod/_remote_module_non_scriptable.py 2022-08-17T12:40:20.7199322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dvt4w68 2022-08-17T12:40:20.7201323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dvt4w68/_remote_module_non_scriptable.py 2022-08-17T12:40:20.7673615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aprnsfo 2022-08-17T12:40:20.7674496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aprnsfo/_remote_module_non_scriptable.py 2022-08-17T12:40:20.9796947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:20.9807357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:20.9832941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:21.0322603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:21.3102638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:21.3204502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:21.3206372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:21.3207547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:21.3211397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:21.3212781Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:21.3214606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:21.3215860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:21.8688993Z ok (2.679s) 2022-08-17T12:40:21.8689232Z 2022-08-17T12:40:21.8689615Z ---------------------------------------------------------------------- 2022-08-17T12:40:21.8689920Z Ran 1 test in 2.679s 2022-08-17T12:40:21.8690093Z 2022-08-17T12:40:21.8690150Z OK 2022-08-17T12:40:21.8690246Z 2022-08-17T12:40:21.8690342Z Generating XML reports... 2022-08-17T12:40:21.8723442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124019.xml 2022-08-17T12:40:23.1866621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:23.1867029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:23.1869352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:23.1869840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:23.3391509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30xmod9s 2022-08-17T12:40:23.3394137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30xmod9s/_remote_module_non_scriptable.py 2022-08-17T12:40:23.6019919Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:23.6030576Z 2022-08-17T12:40:23.6030703Z Running tests... 2022-08-17T12:40:23.6031352Z ---------------------------------------------------------------------- 2022-08-17T12:40:23.8703659Z test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10901 2022-08-17T12:40:23.8731487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10902 2022-08-17T12:40:23.8759385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10903 2022-08-17T12:40:23.8790036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10904 2022-08-17T12:40:24.9111208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:24.9111814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:24.9114696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:24.9115336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:24.9116176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:24.9116826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:24.9119254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:24.9119871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:24.9396813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:24.9397434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:24.9400556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:24.9401220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:24.9538862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:24.9539572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:24.9542605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:24.9543256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:25.0731040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxnm5_5f 2022-08-17T12:40:25.0731840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ljmec_j 2022-08-17T12:40:25.0732624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxnm5_5f/_remote_module_non_scriptable.py 2022-08-17T12:40:25.0733380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ljmec_j/_remote_module_non_scriptable.py 2022-08-17T12:40:25.0994780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_30lhj1 2022-08-17T12:40:25.0995536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_30lhj1/_remote_module_non_scriptable.py 2022-08-17T12:40:25.1115094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1npjsba 2022-08-17T12:40:25.1116285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1npjsba/_remote_module_non_scriptable.py 2022-08-17T12:40:25.3355197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:25.3371108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:25.3639292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:25.3744079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:25.5975820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:25.6077426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:25.6179097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:25.6181100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:25.6182530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:25.6183726Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:25.6184776Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:25.6188524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:26.1853266Z ok (2.582s) 2022-08-17T12:40:26.1859271Z 2022-08-17T12:40:26.1860073Z ---------------------------------------------------------------------- 2022-08-17T12:40:26.1860512Z Ran 1 test in 2.582s 2022-08-17T12:40:26.1860694Z 2022-08-17T12:40:26.1860802Z OK 2022-08-17T12:40:26.1861691Z 2022-08-17T12:40:26.1862354Z Generating XML reports... 2022-08-17T12:40:26.1891096Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124023.xml 2022-08-17T12:40:27.4828902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:27.4829303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:27.4831828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:27.4832190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:27.6309683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacqkjuor 2022-08-17T12:40:27.6312766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacqkjuor/_remote_module_non_scriptable.py 2022-08-17T12:40:27.8925174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:27.8935565Z 2022-08-17T12:40:27.8935678Z Running tests... 2022-08-17T12:40:27.8936103Z ---------------------------------------------------------------------- 2022-08-17T12:40:28.1436613Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.250s) 2022-08-17T12:40:28.1437532Z 2022-08-17T12:40:28.1437830Z ---------------------------------------------------------------------- 2022-08-17T12:40:28.1438109Z Ran 1 test in 0.250s 2022-08-17T12:40:28.1438229Z 2022-08-17T12:40:28.1438305Z OK (skipped=1) 2022-08-17T12:40:28.1438460Z 2022-08-17T12:40:28.1438567Z Generating XML reports... 2022-08-17T12:40:28.1461838Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124027.xml 2022-08-17T12:40:29.3529001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:29.3529408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:29.3532636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:29.3533011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:29.5014988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c83ty69 2022-08-17T12:40:29.5018345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c83ty69/_remote_module_non_scriptable.py 2022-08-17T12:40:29.7634752Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:29.7645522Z 2022-08-17T12:40:29.7645670Z Running tests... 2022-08-17T12:40:29.7646336Z ---------------------------------------------------------------------- 2022-08-17T12:40:30.0220964Z test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11166 2022-08-17T12:40:30.0248609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11167 2022-08-17T12:40:30.0276421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11168 2022-08-17T12:40:30.0312315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11169 2022-08-17T12:40:31.0885000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:31.0885665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:31.0888668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:31.0889302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:31.0924266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:31.0924955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:31.0929248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:31.0929831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:31.1163957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:31.1164367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:31.1166867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:31.1167532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:31.1168294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:31.1168917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:31.1171015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:31.1171628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:31.2519305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd83oywo 2022-08-17T12:40:31.2520308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd83oywo/_remote_module_non_scriptable.py 2022-08-17T12:40:31.2563344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9xtz7ft 2022-08-17T12:40:31.2564710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9xtz7ft/_remote_module_non_scriptable.py 2022-08-17T12:40:31.2751984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppchlu00y 2022-08-17T12:40:31.2752846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppchlu00y/_remote_module_non_scriptable.py 2022-08-17T12:40:31.2762299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1hkk_t9 2022-08-17T12:40:31.2764672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1hkk_t9/_remote_module_non_scriptable.py 2022-08-17T12:40:31.5117007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:31.5129586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:31.5328950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:31.5347730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:32.1371442Z ok (2.372s) 2022-08-17T12:40:32.1371718Z 2022-08-17T12:40:32.1372247Z ---------------------------------------------------------------------- 2022-08-17T12:40:32.1372550Z Ran 1 test in 2.372s 2022-08-17T12:40:32.1372670Z 2022-08-17T12:40:32.1372742Z OK 2022-08-17T12:40:32.1372839Z 2022-08-17T12:40:32.1372937Z Generating XML reports... 2022-08-17T12:40:32.1405497Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124029.xml 2022-08-17T12:40:33.4458726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:33.4459415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:33.4461985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:33.4462575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:33.5974486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke8mvco0 2022-08-17T12:40:33.5977799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke8mvco0/_remote_module_non_scriptable.py 2022-08-17T12:40:33.8600452Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:33.8611336Z 2022-08-17T12:40:33.8611436Z Running tests... 2022-08-17T12:40:33.8611911Z ---------------------------------------------------------------------- 2022-08-17T12:40:34.1230700Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11389 2022-08-17T12:40:34.1259183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11390 2022-08-17T12:40:34.1288793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11391 2022-08-17T12:40:34.1318565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11392 2022-08-17T12:40:35.2439983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:35.2440617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:35.2443779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:35.2444874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:35.2478601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:35.2479985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:35.2481693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:35.2482260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:35.2484980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:35.2485623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:35.2487095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:35.2487716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:35.3198508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:35.3199090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:35.3202316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:35.3203001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:35.4100008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7n6ae8m 2022-08-17T12:40:35.4100897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggec1d_e 2022-08-17T12:40:35.4101651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7n6ae8m/_remote_module_non_scriptable.py 2022-08-17T12:40:35.4102380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggec1d_e/_remote_module_non_scriptable.py 2022-08-17T12:40:35.4136953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66t70quo 2022-08-17T12:40:35.4138555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66t70quo/_remote_module_non_scriptable.py 2022-08-17T12:40:35.4794110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5gi4f8o 2022-08-17T12:40:35.4794975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5gi4f8o/_remote_module_non_scriptable.py 2022-08-17T12:40:35.6740497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:35.6741819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:35.6783000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:35.7437227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:35.9636593Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:40:35.9637812Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:40:35.9836839Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:40:35.9838694Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:40:35.9843171Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:40:35.9844638Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:40:35.9846767Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:40:35.9850943Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:40:36.4381358Z ok (2.577s) 2022-08-17T12:40:36.4381643Z 2022-08-17T12:40:36.4382112Z ---------------------------------------------------------------------- 2022-08-17T12:40:36.4382382Z Ran 1 test in 2.577s 2022-08-17T12:40:36.4382509Z 2022-08-17T12:40:36.4382562Z OK 2022-08-17T12:40:36.4382659Z 2022-08-17T12:40:36.4382754Z Generating XML reports... 2022-08-17T12:40:36.4415524Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124033.xml 2022-08-17T12:40:37.7514220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:37.7514618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:37.7517080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:37.7517477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:37.9050101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfww4ix9 2022-08-17T12:40:37.9054128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfww4ix9/_remote_module_non_scriptable.py 2022-08-17T12:40:38.1704228Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:38.1715211Z 2022-08-17T12:40:38.1715352Z Running tests... 2022-08-17T12:40:38.1715810Z ---------------------------------------------------------------------- 2022-08-17T12:40:38.4395848Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11644 2022-08-17T12:40:38.4423291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11645 2022-08-17T12:40:38.4454346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11646 2022-08-17T12:40:38.4484080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11647 2022-08-17T12:40:39.5347271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:39.5347922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:39.5350912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:39.5512429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:39.5524833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:39.5525507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:39.5526679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:39.5527374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:39.5606816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:39.5607469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:39.5610911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:39.5611591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:39.6381591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:39.6382207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:39.6385107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:39.6386280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:39.6953723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qtji8y2 2022-08-17T12:40:39.6954796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qtji8y2/_remote_module_non_scriptable.py 2022-08-17T12:40:39.7122266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2tc5kfj 2022-08-17T12:40:39.7123398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2tc5kfj/_remote_module_non_scriptable.py 2022-08-17T12:40:39.7207977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8v4c_9h 2022-08-17T12:40:39.7209269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8v4c_9h/_remote_module_non_scriptable.py 2022-08-17T12:40:39.7962901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxra14l6_ 2022-08-17T12:40:39.7963652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxra14l6_/_remote_module_non_scriptable.py 2022-08-17T12:40:39.9602824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:39.9768716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:39.9861744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:40.0600108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:40.3361526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:40.3461785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:40.3564019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:40.3567206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:40.3569139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:40.3571020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:40.3572104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:40.3573698Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:40.9549245Z ok (2.783s) 2022-08-17T12:40:40.9549533Z 2022-08-17T12:40:40.9550006Z ---------------------------------------------------------------------- 2022-08-17T12:40:40.9550435Z Ran 1 test in 2.783s 2022-08-17T12:40:40.9550608Z 2022-08-17T12:40:40.9550712Z OK 2022-08-17T12:40:40.9550879Z 2022-08-17T12:40:40.9551027Z Generating XML reports... 2022-08-17T12:40:40.9584687Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124038.xml 2022-08-17T12:40:42.2432091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:42.2432501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:42.2434852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:42.2435225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:42.3936064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdenhhely 2022-08-17T12:40:42.3939469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdenhhely/_remote_module_non_scriptable.py 2022-08-17T12:40:42.6573723Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:42.6583592Z 2022-08-17T12:40:42.6583739Z Running tests... 2022-08-17T12:40:42.6584208Z ---------------------------------------------------------------------- 2022-08-17T12:40:42.9220247Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11899 2022-08-17T12:40:42.9247779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11900 2022-08-17T12:40:42.9276437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11901 2022-08-17T12:40:42.9306624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11902 2022-08-17T12:40:44.0177491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:44.0178193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:44.0181546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:44.0182244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:44.0326120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:44.0326828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:44.0329180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:44.0329827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:44.0612126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:44.0612747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:44.0615636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:44.0616439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:44.0804434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:44.0804985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:44.0807815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:44.0808367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:44.1773578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84hlsmtd 2022-08-17T12:40:44.1774398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84hlsmtd/_remote_module_non_scriptable.py 2022-08-17T12:40:44.1926255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjul6z8o 2022-08-17T12:40:44.1927425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjul6z8o/_remote_module_non_scriptable.py 2022-08-17T12:40:44.2192006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl19o6udu 2022-08-17T12:40:44.2192893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl19o6udu/_remote_module_non_scriptable.py 2022-08-17T12:40:44.2394580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33h5ixf7 2022-08-17T12:40:44.2395380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33h5ixf7/_remote_module_non_scriptable.py 2022-08-17T12:40:44.4411416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:44.4588782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:44.4830506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:44.5027511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:44.7936950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:44.8040194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:44.8140627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:44.8144187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:44.8147726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:44.8149250Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:44.8150516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:44.8152286Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:45.3371171Z ok (2.678s) 2022-08-17T12:40:45.3371435Z 2022-08-17T12:40:45.3372188Z ---------------------------------------------------------------------- 2022-08-17T12:40:45.3372481Z Ran 1 test in 2.679s 2022-08-17T12:40:45.3372605Z 2022-08-17T12:40:45.3372670Z OK 2022-08-17T12:40:45.3372769Z 2022-08-17T12:40:45.3372850Z Generating XML reports... 2022-08-17T12:40:45.3405727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124042.xml 2022-08-17T12:40:46.6517714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:46.6518106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:46.6520666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:46.6521221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:46.8085542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpgar0_5 2022-08-17T12:40:46.8088993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpgar0_5/_remote_module_non_scriptable.py 2022-08-17T12:40:47.0744666Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:47.0755451Z 2022-08-17T12:40:47.0755582Z Running tests... 2022-08-17T12:40:47.0756213Z ---------------------------------------------------------------------- 2022-08-17T12:40:47.3396267Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12154 2022-08-17T12:40:47.3424484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12155 2022-08-17T12:40:47.3452884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12156 2022-08-17T12:40:47.3486154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12157 2022-08-17T12:40:48.4262053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:48.4262704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:48.4265238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:48.4265899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:48.4329345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:48.4329971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:48.4333477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:48.4334463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:48.4578063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:48.4578769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:48.4581415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:48.4582106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:48.4596133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:48.4596765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:48.4600192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:48.4600873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:48.5860082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm55qytlv 2022-08-17T12:40:48.5861223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm55qytlv/_remote_module_non_scriptable.py 2022-08-17T12:40:48.5912407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproedt1y3 2022-08-17T12:40:48.5913977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproedt1y3/_remote_module_non_scriptable.py 2022-08-17T12:40:48.6155370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57p8pcj1 2022-08-17T12:40:48.6156603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57p8pcj1/_remote_module_non_scriptable.py 2022-08-17T12:40:48.6169417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ifh0w_o 2022-08-17T12:40:48.6171734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ifh0w_o/_remote_module_non_scriptable.py 2022-08-17T12:40:48.8494529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:48.8512475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:48.8828457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:48.8984039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:49.1787138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:49.1888217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:49.2002011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:49.2002742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:49.2003803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:49.2004746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:49.2005654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:49.2006765Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:49.7554156Z ok (2.679s) 2022-08-17T12:40:49.7554388Z 2022-08-17T12:40:49.7554913Z ---------------------------------------------------------------------- 2022-08-17T12:40:49.7555381Z Ran 1 test in 2.680s 2022-08-17T12:40:49.7555603Z 2022-08-17T12:40:49.7555715Z OK 2022-08-17T12:40:49.7556203Z 2022-08-17T12:40:49.7556684Z Generating XML reports... 2022-08-17T12:40:49.7593055Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124047.xml 2022-08-17T12:40:51.0608376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:51.0608778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:51.0611215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:51.0611685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:51.2103215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph09fx_d5 2022-08-17T12:40:51.2106283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph09fx_d5/_remote_module_non_scriptable.py 2022-08-17T12:40:51.4722853Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:51.4733709Z 2022-08-17T12:40:51.4733847Z Running tests... 2022-08-17T12:40:51.4734475Z ---------------------------------------------------------------------- 2022-08-17T12:40:51.7316105Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12389 2022-08-17T12:40:51.7344100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12390 2022-08-17T12:40:51.7372067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12391 2022-08-17T12:40:51.7403857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12392 2022-08-17T12:40:52.7742359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:52.7743020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:52.7746112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:52.7746766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:52.7782023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:52.7782644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:52.7786720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:52.7787370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:52.7892876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:52.7893530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:52.7897437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:52.7898145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:52.7991690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:52.7995098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:52.7995973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:52.7996660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:52.9318576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_g2gy_o 2022-08-17T12:40:52.9320812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_g2gy_o/_remote_module_non_scriptable.py 2022-08-17T12:40:52.9363446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjczkwc9v 2022-08-17T12:40:52.9364745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjczkwc9v/_remote_module_non_scriptable.py 2022-08-17T12:40:52.9480732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_36ytm2 2022-08-17T12:40:52.9482290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_36ytm2/_remote_module_non_scriptable.py 2022-08-17T12:40:52.9571745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8fxr701 2022-08-17T12:40:52.9572993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8fxr701/_remote_module_non_scriptable.py 2022-08-17T12:40:53.1906014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:53.2072209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:53.2175030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:53.2190768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:53.4537822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:53.4539411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:53.4540640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:53.4545832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:53.4547280Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:53.4642755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:53.4644570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:53.4646035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:54.0465032Z ok (2.573s) 2022-08-17T12:40:54.0465295Z 2022-08-17T12:40:54.0465861Z ---------------------------------------------------------------------- 2022-08-17T12:40:54.0466327Z Ran 1 test in 2.573s 2022-08-17T12:40:54.0466450Z 2022-08-17T12:40:54.0466513Z OK 2022-08-17T12:40:54.0466611Z 2022-08-17T12:40:54.0466693Z Generating XML reports... 2022-08-17T12:40:54.0500230Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124051.xml 2022-08-17T12:40:55.3391008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:55.3391394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:55.3394067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:55.3394471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:55.4889400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqztm1ks 2022-08-17T12:40:55.4892292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqztm1ks/_remote_module_non_scriptable.py 2022-08-17T12:40:55.7505695Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:40:55.7516260Z 2022-08-17T12:40:55.7516504Z Running tests... 2022-08-17T12:40:55.7516875Z ---------------------------------------------------------------------- 2022-08-17T12:40:56.0111035Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12624 2022-08-17T12:40:56.0140281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12625 2022-08-17T12:40:56.0170524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12626 2022-08-17T12:40:56.0201025Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12627 2022-08-17T12:40:57.0671322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:57.0671965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:57.0675168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:57.0675849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:57.1235144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:57.1235707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:57.1238948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:57.1239669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:57.1240512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:57.1241421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:57.1246153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:57.1246832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:57.1364839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:57.1365490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:57.1368450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:57.1369125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:57.2264515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh5poozo 2022-08-17T12:40:57.2265607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh5poozo/_remote_module_non_scriptable.py 2022-08-17T12:40:57.2889904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps79pairx 2022-08-17T12:40:57.2891009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps79pairx/_remote_module_non_scriptable.py 2022-08-17T12:40:57.2952028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptj76qw_c 2022-08-17T12:40:57.2953925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptj76qw_c/_remote_module_non_scriptable.py 2022-08-17T12:40:57.3004768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1eui12e 2022-08-17T12:40:57.3006988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1eui12e/_remote_module_non_scriptable.py 2022-08-17T12:40:57.4876830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:40:57.5495272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:40:57.5548786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:40:57.5618624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:40:57.8058770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:40:57.8163667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:40:57.8265842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:40:57.8268205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:40:57.8269579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:57.8270838Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:57.8272052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:57.8363958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:40:58.3263944Z ok (2.574s) 2022-08-17T12:40:58.3264184Z 2022-08-17T12:40:58.3264587Z ---------------------------------------------------------------------- 2022-08-17T12:40:58.3264842Z Ran 1 test in 2.575s 2022-08-17T12:40:58.3264969Z 2022-08-17T12:40:58.3265036Z OK 2022-08-17T12:40:58.3265208Z 2022-08-17T12:40:58.3265305Z Generating XML reports... 2022-08-17T12:40:58.3298664Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124055.xml 2022-08-17T12:40:59.6119110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:40:59.6119527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:40:59.6121374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:40:59.6121734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:40:59.7619312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv5koc73 2022-08-17T12:40:59.7622685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv5koc73/_remote_module_non_scriptable.py 2022-08-17T12:41:00.0237850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:00.0249255Z 2022-08-17T12:41:00.0249349Z Running tests... 2022-08-17T12:41:00.0250083Z ---------------------------------------------------------------------- 2022-08-17T12:41:00.2841275Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12859 2022-08-17T12:41:00.2869070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12860 2022-08-17T12:41:00.2897173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12861 2022-08-17T12:41:00.2928157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12862 2022-08-17T12:41:01.3505593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:01.3506381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:01.3508878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:01.3509554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:01.3643122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:01.3643810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:01.3647247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:01.3647942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:01.4008538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:01.4008987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:01.4012222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:01.4012998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:01.4644485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:01.4645200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:01.4646456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:01.4647062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:01.5115848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpca942zre 2022-08-17T12:41:01.5116722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpca942zre/_remote_module_non_scriptable.py 2022-08-17T12:41:01.5312829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7zhfujh 2022-08-17T12:41:01.5313711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7zhfujh/_remote_module_non_scriptable.py 2022-08-17T12:41:01.5638401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenr1hwy3 2022-08-17T12:41:01.5639433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenr1hwy3/_remote_module_non_scriptable.py 2022-08-17T12:41:01.6216263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuillrxit 2022-08-17T12:41:01.6217073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuillrxit/_remote_module_non_scriptable.py 2022-08-17T12:41:01.7734018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:01.7950097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:01.8301185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:01.8848131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:02.1259178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:02.1359058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:02.1459124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:02.1460346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:02.1461691Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:02.1462956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:02.1464178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:02.1472016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:02.6990704Z ok (2.674s) 2022-08-17T12:41:02.6990939Z 2022-08-17T12:41:02.6991505Z ---------------------------------------------------------------------- 2022-08-17T12:41:02.6991844Z Ran 1 test in 2.674s 2022-08-17T12:41:02.6991966Z 2022-08-17T12:41:02.6992033Z OK 2022-08-17T12:41:02.6992127Z 2022-08-17T12:41:02.6992222Z Generating XML reports... 2022-08-17T12:41:02.7025549Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124100.xml 2022-08-17T12:41:03.9877690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:03.9878086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:03.9880943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:03.9881567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:04.1407796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkg5w61d 2022-08-17T12:41:04.1411021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkg5w61d/_remote_module_non_scriptable.py 2022-08-17T12:41:04.4021122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:04.4031690Z 2022-08-17T12:41:04.4031789Z Running tests... 2022-08-17T12:41:04.4032382Z ---------------------------------------------------------------------- 2022-08-17T12:41:04.6632492Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13094 2022-08-17T12:41:04.6659965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13095 2022-08-17T12:41:04.6688171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13096 2022-08-17T12:41:04.6719021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13097 2022-08-17T12:41:05.6455978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:05.6456669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:05.6459109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:05.6459789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:05.6754124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:05.6754753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:05.6757765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:05.6758402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:05.7085320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:05.7086219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:05.7089406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:05.7090090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:05.7149182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:05.7149815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:05.7153043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:05.7153737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:05.8040604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzydtn5oz 2022-08-17T12:41:05.8041583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzydtn5oz/_remote_module_non_scriptable.py 2022-08-17T12:41:05.8321333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47tv_vsi 2022-08-17T12:41:05.8322087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47tv_vsi/_remote_module_non_scriptable.py 2022-08-17T12:41:05.8676146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yfsdgop 2022-08-17T12:41:05.8677714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yfsdgop/_remote_module_non_scriptable.py 2022-08-17T12:41:05.8724996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza5oxezy 2022-08-17T12:41:05.8726730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza5oxezy/_remote_module_non_scriptable.py 2022-08-17T12:41:06.0622532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:06.0899846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:06.1279460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:06.1323442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:06.4197368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:06.4297859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:06.4399499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:06.4400811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:06.4401782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:06.4403259Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:06.4404139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:06.4404889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:06.9781691Z ok (2.575s) 2022-08-17T12:41:06.9781954Z 2022-08-17T12:41:06.9782507Z ---------------------------------------------------------------------- 2022-08-17T12:41:06.9782833Z Ran 1 test in 2.575s 2022-08-17T12:41:06.9782958Z 2022-08-17T12:41:06.9783022Z OK 2022-08-17T12:41:06.9783134Z 2022-08-17T12:41:06.9783230Z Generating XML reports... 2022-08-17T12:41:06.9819299Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124104.xml 2022-08-17T12:41:08.2914609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:08.2915001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:08.2917438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:08.2917817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:08.4416662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdfus503 2022-08-17T12:41:08.4420163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdfus503/_remote_module_non_scriptable.py 2022-08-17T12:41:08.7033603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:08.7044210Z 2022-08-17T12:41:08.7044343Z Running tests... 2022-08-17T12:41:08.7045088Z ---------------------------------------------------------------------- 2022-08-17T12:41:08.9674226Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13329 2022-08-17T12:41:08.9701716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13330 2022-08-17T12:41:08.9730724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13331 2022-08-17T12:41:08.9762448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13332 2022-08-17T12:41:10.0474099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:10.0474823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:10.0478141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:10.0478776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:10.0771811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:10.0772483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:10.0773765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:10.0774406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:10.0959325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:10.0959990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:10.0963197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:10.0963820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:10.1037107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:10.1038098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:10.1041906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:10.1042473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:10.2069871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2i5k2cy 2022-08-17T12:41:10.2070975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2i5k2cy/_remote_module_non_scriptable.py 2022-08-17T12:41:10.2373423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6xu0dd0 2022-08-17T12:41:10.2374525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6xu0dd0/_remote_module_non_scriptable.py 2022-08-17T12:41:10.2573183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflz6cp6x 2022-08-17T12:41:10.2574041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflz6cp6x/_remote_module_non_scriptable.py 2022-08-17T12:41:10.2621823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptprx572s 2022-08-17T12:41:10.2622667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptprx572s/_remote_module_non_scriptable.py 2022-08-17T12:41:10.4700645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:10.4990379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:10.5207483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:10.5209945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:10.8244062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:10.8245012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:10.8346703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:10.8349940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:10.8351697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:10.8353437Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:10.8449169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:10.8454015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:11.3826813Z ok (2.678s) 2022-08-17T12:41:11.3827064Z 2022-08-17T12:41:11.3827415Z ---------------------------------------------------------------------- 2022-08-17T12:41:11.3827680Z Ran 1 test in 2.678s 2022-08-17T12:41:11.3827812Z 2022-08-17T12:41:11.3827876Z OK 2022-08-17T12:41:11.3827973Z 2022-08-17T12:41:11.3828055Z Generating XML reports... 2022-08-17T12:41:11.3861272Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124108.xml 2022-08-17T12:41:12.6876291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:12.6876694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:12.6878819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:12.6879224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:12.8408780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjdkjk_q 2022-08-17T12:41:12.8412352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjdkjk_q/_remote_module_non_scriptable.py 2022-08-17T12:41:13.1100948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:13.1111579Z 2022-08-17T12:41:13.1111689Z Running tests... 2022-08-17T12:41:13.1112282Z ---------------------------------------------------------------------- 2022-08-17T12:41:13.3712378Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13564 2022-08-17T12:41:13.3739632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13565 2022-08-17T12:41:13.3768748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13566 2022-08-17T12:41:13.3798926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13567 2022-08-17T12:41:14.4200453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:14.4201118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:14.4203631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:14.4204283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:14.4359795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:14.4360479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:14.4363642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:14.4364316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:14.4567933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:14.4568579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:14.4571861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:14.4572526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:14.4764999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:14.4765708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:14.4768611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:14.4769697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:14.5786516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcl3c1ln 2022-08-17T12:41:14.5787516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcl3c1ln/_remote_module_non_scriptable.py 2022-08-17T12:41:14.5947584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk4t3sy1 2022-08-17T12:41:14.5948479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk4t3sy1/_remote_module_non_scriptable.py 2022-08-17T12:41:14.6144933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_uxl3e1 2022-08-17T12:41:14.6146302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_uxl3e1/_remote_module_non_scriptable.py 2022-08-17T12:41:14.6314503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts58lxfr 2022-08-17T12:41:14.6315550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts58lxfr/_remote_module_non_scriptable.py 2022-08-17T12:41:14.8390874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:14.8545181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:14.8768311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:14.8887312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:15.1760778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:15.1761705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:15.1861296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:15.1866311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:15.1867528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:15.1868652Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:15.1869763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:15.1870985Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:15.7863106Z ok (2.675s) 2022-08-17T12:41:15.7863287Z 2022-08-17T12:41:15.7863731Z ---------------------------------------------------------------------- 2022-08-17T12:41:15.7864226Z Ran 1 test in 2.675s 2022-08-17T12:41:15.7864391Z 2022-08-17T12:41:15.7864455Z OK 2022-08-17T12:41:15.7864572Z 2022-08-17T12:41:15.7864669Z Generating XML reports... 2022-08-17T12:41:15.7897427Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124113.xml 2022-08-17T12:41:17.0966839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:17.0967218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:17.0969865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:17.0970304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:17.2463729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7orsggcr 2022-08-17T12:41:17.2467289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7orsggcr/_remote_module_non_scriptable.py 2022-08-17T12:41:17.5081127Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:17.5092067Z 2022-08-17T12:41:17.5092440Z Running tests... 2022-08-17T12:41:17.5092833Z ---------------------------------------------------------------------- 2022-08-17T12:41:17.7683025Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13799 2022-08-17T12:41:17.7711631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13800 2022-08-17T12:41:17.7739813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13801 2022-08-17T12:41:17.7770335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13802 2022-08-17T12:41:18.8037100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:18.8037730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:18.8040871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:18.8164189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:18.8165424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:18.8166249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:18.8168642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:18.8169253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:18.8371230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:18.8371882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:18.8375186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:18.8375873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:18.8606456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:18.8607044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:18.8609903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:18.8610464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:18.9639365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd4x_4p0 2022-08-17T12:41:18.9640232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd4x_4p0/_remote_module_non_scriptable.py 2022-08-17T12:41:18.9746520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1xg6ung 2022-08-17T12:41:18.9747773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1xg6ung/_remote_module_non_scriptable.py 2022-08-17T12:41:18.9958199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq9nr2t_ 2022-08-17T12:41:18.9959001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq9nr2t_/_remote_module_non_scriptable.py 2022-08-17T12:41:19.0161181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsgnh3dr 2022-08-17T12:41:19.0162091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsgnh3dr/_remote_module_non_scriptable.py 2022-08-17T12:41:19.2356730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:19.2499445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:19.2576957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:19.2758646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:19.5398264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:19.5500906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:19.5604382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:19.5605309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:19.5606848Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:19.5608103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:19.5609355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:19.5703807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:20.1834109Z ok (2.674s) 2022-08-17T12:41:20.1834358Z 2022-08-17T12:41:20.1835157Z ---------------------------------------------------------------------- 2022-08-17T12:41:20.1835439Z Ran 1 test in 2.674s 2022-08-17T12:41:20.1835560Z 2022-08-17T12:41:20.1835625Z OK 2022-08-17T12:41:20.1835722Z 2022-08-17T12:41:20.1835817Z Generating XML reports... 2022-08-17T12:41:20.1878024Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124117.xml 2022-08-17T12:41:21.4970436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:21.4971134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:21.4973665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:21.4974271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:21.6496754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqcpha10 2022-08-17T12:41:21.6500865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqcpha10/_remote_module_non_scriptable.py 2022-08-17T12:41:21.9153012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:21.9164318Z 2022-08-17T12:41:21.9164603Z Running tests... 2022-08-17T12:41:22.1764892Z ---------------------------------------------------------------------- 2022-08-17T12:41:22.1765721Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14034 2022-08-17T12:41:22.1795135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14035 2022-08-17T12:41:22.1825007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14036 2022-08-17T12:41:22.1858460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14037 2022-08-17T12:41:23.1900914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:23.1901577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:23.1903822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:23.1904455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:23.1995607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:23.1996276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:23.1997075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:23.1997979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:23.1999501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:23.2000119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:23.2001061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:23.2002387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:23.2080160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:23.2080835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:23.2084164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:23.2084876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:23.3521474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eiy3dqu 2022-08-17T12:41:23.3522591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eiy3dqu/_remote_module_non_scriptable.py 2022-08-17T12:41:23.3588163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxk17zjl 2022-08-17T12:41:23.3589480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxk17zjl/_remote_module_non_scriptable.py 2022-08-17T12:41:23.3637514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6y2ubir 2022-08-17T12:41:23.3639195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6y2ubir/_remote_module_non_scriptable.py 2022-08-17T12:41:23.3851707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8_jjvdt 2022-08-17T12:41:23.3852849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8_jjvdt/_remote_module_non_scriptable.py 2022-08-17T12:41:23.6123287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:23.6176619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:23.6246250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:23.6511565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:24.3918036Z ok (2.475s) 2022-08-17T12:41:24.3918199Z 2022-08-17T12:41:24.3918572Z ---------------------------------------------------------------------- 2022-08-17T12:41:24.3918856Z Ran 1 test in 2.475s 2022-08-17T12:41:24.3918968Z 2022-08-17T12:41:24.3919031Z OK 2022-08-17T12:41:24.3919127Z 2022-08-17T12:41:24.3919223Z Generating XML reports... 2022-08-17T12:41:24.3954254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124121.xml 2022-08-17T12:41:25.7280483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:25.7280902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:25.7283865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:25.7284226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:25.8837512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_ut4qyv 2022-08-17T12:41:25.8841467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_ut4qyv/_remote_module_non_scriptable.py 2022-08-17T12:41:26.1507393Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:26.1519490Z 2022-08-17T12:41:26.1519791Z Running tests... 2022-08-17T12:41:26.1520441Z ---------------------------------------------------------------------- 2022-08-17T12:41:26.4196814Z test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14277 2022-08-17T12:41:26.4225037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14278 2022-08-17T12:41:26.4254012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14279 2022-08-17T12:41:26.4286451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14280 2022-08-17T12:41:27.4807417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:27.4808074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:27.4811360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:27.4812045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:27.5399802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:27.5400789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:27.5403274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:27.5403917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:27.5477899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:27.5478569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:27.5482115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:27.5482748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:27.5575989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:27.5576650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:27.5579935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:27.5580576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:27.6401457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun7a2p1x 2022-08-17T12:41:27.6402702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun7a2p1x/_remote_module_non_scriptable.py 2022-08-17T12:41:27.6995403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7aqpxg9r 2022-08-17T12:41:27.6996198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7aqpxg9r/_remote_module_non_scriptable.py 2022-08-17T12:41:27.7051492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79o2umsm 2022-08-17T12:41:27.7052836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79o2umsm/_remote_module_non_scriptable.py 2022-08-17T12:41:27.7146324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0_ljq3u 2022-08-17T12:41:27.7147976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0_ljq3u/_remote_module_non_scriptable.py 2022-08-17T12:41:27.9044188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:27.9600046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:27.9624828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:27.9752657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:29.4362124Z ok (3.284s) 2022-08-17T12:41:29.4362433Z 2022-08-17T12:41:29.4362971Z ---------------------------------------------------------------------- 2022-08-17T12:41:29.4363352Z Ran 1 test in 3.284s 2022-08-17T12:41:29.4363473Z 2022-08-17T12:41:29.4363541Z OK 2022-08-17T12:41:29.4363624Z 2022-08-17T12:41:29.4363733Z Generating XML reports... 2022-08-17T12:41:29.4399541Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124126.xml 2022-08-17T12:41:30.7610204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:30.7610590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:30.7613535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:30.7614080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:30.9135630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_qentf9 2022-08-17T12:41:30.9139320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_qentf9/_remote_module_non_scriptable.py 2022-08-17T12:41:31.1773163Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:31.1784756Z 2022-08-17T12:41:31.1785206Z Running tests... 2022-08-17T12:41:31.1785597Z ---------------------------------------------------------------------- 2022-08-17T12:41:31.4429544Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14520 2022-08-17T12:41:31.4458087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14521 2022-08-17T12:41:31.4486824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14522 2022-08-17T12:41:31.4517228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14523 2022-08-17T12:41:32.5148963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:32.5149564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:32.5152987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:32.5153617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:32.5158614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:32.5159248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:32.5163593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:32.5164251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:32.5280673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:32.5281307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:32.5284541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:32.5285171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:32.5478267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:32.5478951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:32.5482285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:32.5482879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:32.6739394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuisjz48 2022-08-17T12:41:32.6740770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuisjz48/_remote_module_non_scriptable.py 2022-08-17T12:41:32.6763835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_scjfje5 2022-08-17T12:41:32.6765480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_scjfje5/_remote_module_non_scriptable.py 2022-08-17T12:41:32.6872103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0hgmp9q 2022-08-17T12:41:32.6872957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0hgmp9q/_remote_module_non_scriptable.py 2022-08-17T12:41:32.7078982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaev4f13c 2022-08-17T12:41:32.7079798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaev4f13c/_remote_module_non_scriptable.py 2022-08-17T12:41:32.9393577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:32.9489668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:32.9656029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:32.9766025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:33.2446366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:33.2447319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:33.2548035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:33.2550689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:33.2552062Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:33.2553145Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:33.2554335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:33.2555572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:33.5268822Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #150 to worker3, as the agent is shutting down 2022-08-17T12:41:33.5269753Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5270724Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5271730Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5272780Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:41:33.5273693Z [W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #151 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:41:33.5274418Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:41:33.5275112Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:41:33.5280999Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5306808Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5307491Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5340849Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #152 to worker0, as the agent is shutting down 2022-08-17T12:41:33.5347980Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #150 to worker1, as the agent is shutting down 2022-08-17T12:41:33.5370050Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5371230Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5373313Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5473997Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.5475166Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:41:33.9581475Z ok (2.779s) 2022-08-17T12:41:33.9581716Z 2022-08-17T12:41:33.9582042Z ---------------------------------------------------------------------- 2022-08-17T12:41:33.9582295Z Ran 1 test in 2.780s 2022-08-17T12:41:33.9582415Z 2022-08-17T12:41:33.9582481Z OK 2022-08-17T12:41:33.9582576Z 2022-08-17T12:41:33.9582672Z Generating XML reports... 2022-08-17T12:41:33.9618318Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124131.xml 2022-08-17T12:41:35.2443074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:35.2443482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:35.2446413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:35.2446850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:35.3975746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5nij403 2022-08-17T12:41:35.3978942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5nij403/_remote_module_non_scriptable.py 2022-08-17T12:41:35.6603541Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:35.6614590Z 2022-08-17T12:41:35.6614719Z Running tests... 2022-08-17T12:41:35.6615310Z ---------------------------------------------------------------------- 2022-08-17T12:41:35.9264397Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14775 2022-08-17T12:41:35.9293646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14776 2022-08-17T12:41:35.9322505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14777 2022-08-17T12:41:35.9353376Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14778 2022-08-17T12:41:36.9719772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:36.9720439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:36.9723559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:36.9724240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:36.9762267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:36.9762880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:36.9766970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:36.9767626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:36.9777675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:36.9778271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:36.9782773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:36.9783453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:36.9902916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:36.9903550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:36.9907298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:36.9907902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:37.1306586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchc3za0d 2022-08-17T12:41:37.1307349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchc3za0d/_remote_module_non_scriptable.py 2022-08-17T12:41:37.1341512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxy3e1ppz 2022-08-17T12:41:37.1343171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxy3e1ppz/_remote_module_non_scriptable.py 2022-08-17T12:41:37.1377855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyee926_6 2022-08-17T12:41:37.1379652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyee926_6/_remote_module_non_scriptable.py 2022-08-17T12:41:37.1466476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl13knvbf 2022-08-17T12:41:37.1469368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl13knvbf/_remote_module_non_scriptable.py 2022-08-17T12:41:37.3919650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:37.3938827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:37.4002551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:37.4052069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:38.0411768Z ok (2.379s) 2022-08-17T12:41:38.0412001Z 2022-08-17T12:41:38.0412391Z ---------------------------------------------------------------------- 2022-08-17T12:41:38.0412656Z Ran 1 test in 2.380s 2022-08-17T12:41:38.0412779Z 2022-08-17T12:41:38.0412843Z OK 2022-08-17T12:41:38.0412978Z 2022-08-17T12:41:38.0413086Z Generating XML reports... 2022-08-17T12:41:38.0446189Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124135.xml 2022-08-17T12:41:39.3366978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:39.3367633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:39.3370317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:39.3370702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:39.4858553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquz0dbb2 2022-08-17T12:41:39.4861317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquz0dbb2/_remote_module_non_scriptable.py 2022-08-17T12:41:39.7476817Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:39.7487305Z 2022-08-17T12:41:39.7487789Z Running tests... 2022-08-17T12:41:39.7495677Z ---------------------------------------------------------------------- 2022-08-17T12:41:40.0078762Z test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15018 2022-08-17T12:41:40.0107424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15019 2022-08-17T12:41:40.0136165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15020 2022-08-17T12:41:40.0167402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15021 2022-08-17T12:41:41.0943004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:41.0943686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:41.0946548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:41.0947208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:41.1245049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:41.1245714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:41.1248011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:41.1248693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:41.1548718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:41.1549381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:41.1551864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:41.1552491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:41.1574278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:41.1574954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:41.1579429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:41.1580001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:41.2580562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge_5z7vs 2022-08-17T12:41:41.2581830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge_5z7vs/_remote_module_non_scriptable.py 2022-08-17T12:41:41.2821529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk9rdwvn 2022-08-17T12:41:41.2822373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk9rdwvn/_remote_module_non_scriptable.py 2022-08-17T12:41:41.3131906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjpxvydp 2022-08-17T12:41:41.3132659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjpxvydp/_remote_module_non_scriptable.py 2022-08-17T12:41:41.3170047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmee1c3_k 2022-08-17T12:41:41.3171781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmee1c3_k/_remote_module_non_scriptable.py 2022-08-17T12:41:41.5198130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:41.5427489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:41.5751792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:41.5762596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:42.2228920Z ok (2.474s) 2022-08-17T12:41:42.2229184Z 2022-08-17T12:41:42.2229740Z ---------------------------------------------------------------------- 2022-08-17T12:41:42.2230075Z Ran 1 test in 2.474s 2022-08-17T12:41:42.2230293Z 2022-08-17T12:41:42.2230409Z OK 2022-08-17T12:41:42.2230568Z 2022-08-17T12:41:42.2230739Z Generating XML reports... 2022-08-17T12:41:42.2263579Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124139.xml 2022-08-17T12:41:43.5111686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:43.5112123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:43.5114669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:43.5115082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:43.6606623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidytw5o8 2022-08-17T12:41:43.6609613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidytw5o8/_remote_module_non_scriptable.py 2022-08-17T12:41:43.9218526Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:43.9228536Z 2022-08-17T12:41:43.9228679Z Running tests... 2022-08-17T12:41:43.9229115Z ---------------------------------------------------------------------- 2022-08-17T12:41:43.9254288Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest) 2022-08-17T12:41:44.1859094Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15241 2022-08-17T12:41:44.1887950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15242 2022-08-17T12:41:44.1941566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15243 2022-08-17T12:41:44.1970357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15244 2022-08-17T12:41:45.2687743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:45.2688455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:45.2689305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:45.2689946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:45.2749963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:45.2750648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:45.2754934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:45.2755605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:45.3038292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:45.3038960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:45.3042129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:45.3042796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:45.3225225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:45.3225755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:45.3228869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:45.3229566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:45.4274512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ev56ipl 2022-08-17T12:41:45.4275591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ev56ipl/_remote_module_non_scriptable.py 2022-08-17T12:41:45.4340272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbjz5bqp 2022-08-17T12:41:45.4341514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbjz5bqp/_remote_module_non_scriptable.py 2022-08-17T12:41:45.4648690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco9h78x1 2022-08-17T12:41:45.4649579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco9h78x1/_remote_module_non_scriptable.py 2022-08-17T12:41:45.4808952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12f1f886 2022-08-17T12:41:45.4810403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12f1f886/_remote_module_non_scriptable.py 2022-08-17T12:41:45.6890508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:45.6929931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:45.7297550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:45.7449824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:46.4030743Z ok (2.480s) 2022-08-17T12:41:46.4030999Z 2022-08-17T12:41:46.4031565Z ---------------------------------------------------------------------- 2022-08-17T12:41:46.4031850Z Ran 1 test in 2.480s 2022-08-17T12:41:46.4031959Z 2022-08-17T12:41:46.4032024Z OK 2022-08-17T12:41:46.4032121Z 2022-08-17T12:41:46.4032218Z Generating XML reports... 2022-08-17T12:41:46.4066155Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124143.xml 2022-08-17T12:41:47.7106522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:47.7107024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:47.7109303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:47.7109701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:47.8638660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprqyrze1 2022-08-17T12:41:47.8641823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprqyrze1/_remote_module_non_scriptable.py 2022-08-17T12:41:48.1293738Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:48.1304532Z 2022-08-17T12:41:48.1304631Z Running tests... 2022-08-17T12:41:48.1305127Z ---------------------------------------------------------------------- 2022-08-17T12:41:48.3996739Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15484 2022-08-17T12:41:48.4023984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15485 2022-08-17T12:41:48.4054105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15486 2022-08-17T12:41:48.4083156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15487 2022-08-17T12:41:49.3809120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:49.3809794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:49.3813269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:49.3813944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:49.3978356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:49.3978958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:49.3982446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:49.3983085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:49.4278696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:49.4279554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:49.4281536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:49.4282072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:49.4516795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:49.4517473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:49.4518946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:49.4519626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:49.5418108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4tnkwxg 2022-08-17T12:41:49.5419756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4tnkwxg/_remote_module_non_scriptable.py 2022-08-17T12:41:49.5576286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0aim3wn 2022-08-17T12:41:49.5577087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0aim3wn/_remote_module_non_scriptable.py 2022-08-17T12:41:49.5899529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64re29wf 2022-08-17T12:41:49.5900298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64re29wf/_remote_module_non_scriptable.py 2022-08-17T12:41:49.6123725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptef8bzzv 2022-08-17T12:41:49.6124624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptef8bzzv/_remote_module_non_scriptable.py 2022-08-17T12:41:49.8048019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:49.8194693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:49.8551317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:49.8761822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:50.1194578Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:41:50.1195793Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:41:50.1397294Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:41:50.1398816Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:41:50.1400414Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:41:50.1401496Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:41:50.1407550Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2322: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`. 2022-08-17T12:41:50.1408685Z "Argument order of nn.functional.embedding_bag was changed. " 2022-08-17T12:41:50.6144339Z ok (2.484s) 2022-08-17T12:41:50.6144570Z 2022-08-17T12:41:50.6145117Z ---------------------------------------------------------------------- 2022-08-17T12:41:50.6145497Z Ran 1 test in 2.484s 2022-08-17T12:41:50.6145617Z 2022-08-17T12:41:50.6145681Z OK 2022-08-17T12:41:50.6146007Z 2022-08-17T12:41:50.6146107Z Generating XML reports... 2022-08-17T12:41:50.6179979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124148.xml 2022-08-17T12:41:51.9214475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:51.9214879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:51.9216716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:51.9217077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:52.0739076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb3yswoy 2022-08-17T12:41:52.0742312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb3yswoy/_remote_module_non_scriptable.py 2022-08-17T12:41:52.3424506Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:52.3435633Z 2022-08-17T12:41:52.3435761Z Running tests... 2022-08-17T12:41:52.3436363Z ---------------------------------------------------------------------- 2022-08-17T12:41:52.6153999Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15739 2022-08-17T12:41:52.6182945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15740 2022-08-17T12:41:52.6212721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15741 2022-08-17T12:41:52.6244754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15742 2022-08-17T12:41:53.6688827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:53.6689508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:53.6692169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:53.6692839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:53.6696944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:53.6697551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:53.6704476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:53.6705737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:53.7025558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:53.7026210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:53.7028445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:53.7029110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:53.7332201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:53.7332718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:53.7335405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:53.7336061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:53.8302290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmi5jt_8r 2022-08-17T12:41:53.8302964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn5_dq76 2022-08-17T12:41:53.8303842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmi5jt_8r/_remote_module_non_scriptable.py 2022-08-17T12:41:53.8304582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn5_dq76/_remote_module_non_scriptable.py 2022-08-17T12:41:53.8608813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqaszxtv 2022-08-17T12:41:53.8609980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqaszxtv/_remote_module_non_scriptable.py 2022-08-17T12:41:53.8912718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobcfegy7 2022-08-17T12:41:53.8913545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobcfegy7/_remote_module_non_scriptable.py 2022-08-17T12:41:54.0927716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:54.0935623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:54.1224923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:54.1520427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:54.4375183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:54.4375711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:54.4479197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:54.4480493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:54.4481778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:54.4482731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:54.4486216Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:54.4488025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:55.0307554Z ok (2.687s) 2022-08-17T12:41:55.0307797Z 2022-08-17T12:41:55.0308180Z ---------------------------------------------------------------------- 2022-08-17T12:41:55.0308437Z Ran 1 test in 2.687s 2022-08-17T12:41:55.0308562Z 2022-08-17T12:41:55.0308628Z OK 2022-08-17T12:41:55.0308730Z 2022-08-17T12:41:55.0308847Z Generating XML reports... 2022-08-17T12:41:55.0342693Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124152.xml 2022-08-17T12:41:56.3638530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:56.3638901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:56.3641425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:56.3641808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:56.5199284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wxeb_pf 2022-08-17T12:41:56.5203038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wxeb_pf/_remote_module_non_scriptable.py 2022-08-17T12:41:56.7990165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:41:56.8001697Z 2022-08-17T12:41:56.8002064Z Running tests... 2022-08-17T12:41:56.8002658Z ---------------------------------------------------------------------- 2022-08-17T12:41:57.0668926Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15974 2022-08-17T12:41:57.0698727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15975 2022-08-17T12:41:57.0728717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15976 2022-08-17T12:41:57.0760789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15977 2022-08-17T12:41:58.1710549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:58.1711187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:58.1714509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:58.1715211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:58.1855778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:58.1856388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:58.1859259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:58.1859926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:58.1947350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:58.1947975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:58.1951941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:58.1952633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:58.2119020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:41:58.2119654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:41:58.2122246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:41:58.2122890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:41:58.3334079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z78pjcv 2022-08-17T12:41:58.3335130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z78pjcv/_remote_module_non_scriptable.py 2022-08-17T12:41:58.3456824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph22pgzir 2022-08-17T12:41:58.3458057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph22pgzir/_remote_module_non_scriptable.py 2022-08-17T12:41:58.3547348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp669gnqum 2022-08-17T12:41:58.3548252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp669gnqum/_remote_module_non_scriptable.py 2022-08-17T12:41:58.3712130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1oyvql4 2022-08-17T12:41:58.3712955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1oyvql4/_remote_module_non_scriptable.py 2022-08-17T12:41:58.5961534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:41:58.6074469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:41:58.6148193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:41:58.6315099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:41:58.9065783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:41:58.9067545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:41:58.9068711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:41:58.9072460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:41:58.9073823Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:58.9170394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:58.9172614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:58.9173881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:41:59.4824100Z ok (2.682s) 2022-08-17T12:41:59.4824280Z 2022-08-17T12:41:59.4825195Z ---------------------------------------------------------------------- 2022-08-17T12:41:59.4825586Z Ran 1 test in 2.682s 2022-08-17T12:41:59.4825697Z 2022-08-17T12:41:59.4825779Z OK 2022-08-17T12:41:59.4825886Z 2022-08-17T12:41:59.4825984Z Generating XML reports... 2022-08-17T12:41:59.4859402Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124156.xml 2022-08-17T12:42:00.7881882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:00.7882372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:00.7884388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:00.7884826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:00.9384697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9l1rfgs 2022-08-17T12:42:00.9387920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9l1rfgs/_remote_module_non_scriptable.py 2022-08-17T12:42:01.1997478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:01.2008073Z 2022-08-17T12:42:01.2008191Z Running tests... 2022-08-17T12:42:01.2008610Z ---------------------------------------------------------------------- 2022-08-17T12:42:01.4643709Z test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16209 2022-08-17T12:42:01.4671719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16210 2022-08-17T12:42:01.4700302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16211 2022-08-17T12:42:01.4731312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16212 2022-08-17T12:42:02.5310743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:02.5311432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:02.5314456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:02.5315174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:02.5770088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:02.5770722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:02.5774050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:02.5774706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:02.5871778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:02.5872453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:02.5875527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:02.5876200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:02.6234980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:02.6235693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:02.6238052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:02.6238614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:02.6908024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts8h3suv 2022-08-17T12:42:02.6909097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts8h3suv/_remote_module_non_scriptable.py 2022-08-17T12:42:02.7368648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyv7e33i 2022-08-17T12:42:02.7370410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyv7e33i/_remote_module_non_scriptable.py 2022-08-17T12:42:02.7461440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rqmk0fx 2022-08-17T12:42:02.7462584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rqmk0fx/_remote_module_non_scriptable.py 2022-08-17T12:42:02.7823810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkngsrlh 2022-08-17T12:42:02.7824857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkngsrlh/_remote_module_non_scriptable.py 2022-08-17T12:42:02.9544747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:02.9991267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:03.0059950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:03.0455966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:03.6791748Z ok (2.478s) 2022-08-17T12:42:03.6792023Z 2022-08-17T12:42:03.6792379Z ---------------------------------------------------------------------- 2022-08-17T12:42:03.6792652Z Ran 1 test in 2.478s 2022-08-17T12:42:03.6792775Z 2022-08-17T12:42:03.6792828Z OK 2022-08-17T12:42:03.6792928Z 2022-08-17T12:42:03.6793024Z Generating XML reports... 2022-08-17T12:42:03.6827607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124201.xml 2022-08-17T12:42:04.9985797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:04.9986192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:04.9989245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:04.9989651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:05.1608001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpderz92q5 2022-08-17T12:42:05.1611355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpderz92q5/_remote_module_non_scriptable.py 2022-08-17T12:42:05.4289686Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:05.4300447Z 2022-08-17T12:42:05.4300589Z Running tests... 2022-08-17T12:42:05.4301199Z ---------------------------------------------------------------------- 2022-08-17T12:42:05.6927553Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16452 2022-08-17T12:42:05.6966829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16453 2022-08-17T12:42:05.7004940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16454 2022-08-17T12:42:05.7035397Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16455 2022-08-17T12:42:06.7787118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:06.7787738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:06.7790905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:06.7791560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:06.7896640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:06.7897282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:06.7900508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:06.7901209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:06.8051016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:06.8051670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:06.8054857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:06.8055490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:06.8241361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:06.8241864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:06.8244523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:06.8245643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:06.9385382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27ss50sr 2022-08-17T12:42:06.9386621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27ss50sr/_remote_module_non_scriptable.py 2022-08-17T12:42:06.9506888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxme0kj2 2022-08-17T12:42:06.9508178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxme0kj2/_remote_module_non_scriptable.py 2022-08-17T12:42:06.9657793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiq0eept1 2022-08-17T12:42:06.9658844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiq0eept1/_remote_module_non_scriptable.py 2022-08-17T12:42:06.9848418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchmzc6a_ 2022-08-17T12:42:06.9849441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchmzc6a_/_remote_module_non_scriptable.py 2022-08-17T12:42:07.1980009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:07.2117149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:07.2391833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:07.2519039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:07.9095468Z ok (2.479s) 2022-08-17T12:42:07.9095723Z 2022-08-17T12:42:07.9096263Z ---------------------------------------------------------------------- 2022-08-17T12:42:07.9096660Z Ran 1 test in 2.479s 2022-08-17T12:42:07.9096785Z 2022-08-17T12:42:07.9096836Z OK 2022-08-17T12:42:07.9096937Z 2022-08-17T12:42:07.9097037Z Generating XML reports... 2022-08-17T12:42:07.9130243Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124205.xml 2022-08-17T12:42:09.2112322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:09.2112723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:09.2114981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:09.2115419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:09.3659873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq74heop8 2022-08-17T12:42:09.3663082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq74heop8/_remote_module_non_scriptable.py 2022-08-17T12:42:09.6304532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:09.6314892Z 2022-08-17T12:42:09.6315039Z Running tests... 2022-08-17T12:42:09.6315689Z ---------------------------------------------------------------------- 2022-08-17T12:42:09.8954401Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16675 2022-08-17T12:42:09.8984747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16676 2022-08-17T12:42:09.9014054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16677 2022-08-17T12:42:09.9043854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16678 2022-08-17T12:42:10.9281086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:10.9281758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:10.9284844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:10.9285510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:10.9460192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:10.9460831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:10.9464348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:10.9465005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:10.9470358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:10.9471317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:10.9475239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:10.9475828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:10.9570291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:10.9570947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:10.9574394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:10.9575048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:11.0881627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu9juega 2022-08-17T12:42:11.0883042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu9juega/_remote_module_non_scriptable.py 2022-08-17T12:42:11.1079592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tfdfgee 2022-08-17T12:42:11.1080356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tfdfgee/_remote_module_non_scriptable.py 2022-08-17T12:42:11.1081214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5i69d7mb 2022-08-17T12:42:11.1083890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5i69d7mb/_remote_module_non_scriptable.py 2022-08-17T12:42:11.1171354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tprljp_ 2022-08-17T12:42:11.1172535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tprljp_/_remote_module_non_scriptable.py 2022-08-17T12:42:11.3512753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:11.3737589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:11.3738273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:11.3816017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:12.1105420Z ok (2.479s) 2022-08-17T12:42:12.1105579Z 2022-08-17T12:42:12.1105980Z ---------------------------------------------------------------------- 2022-08-17T12:42:12.1106247Z Ran 1 test in 2.479s 2022-08-17T12:42:12.1106424Z 2022-08-17T12:42:12.1106546Z OK 2022-08-17T12:42:12.1106659Z 2022-08-17T12:42:12.1106755Z Generating XML reports... 2022-08-17T12:42:12.1140367Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124209.xml 2022-08-17T12:42:13.4169302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:13.4170020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:13.4173292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:13.4173879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:13.5730494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ouefxw1 2022-08-17T12:42:13.5734315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ouefxw1/_remote_module_non_scriptable.py 2022-08-17T12:42:13.8386675Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:13.8398184Z 2022-08-17T12:42:13.8398715Z Running tests... 2022-08-17T12:42:13.8399145Z ---------------------------------------------------------------------- 2022-08-17T12:42:14.1041106Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16898 2022-08-17T12:42:14.1069197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16899 2022-08-17T12:42:14.1097810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16900 2022-08-17T12:42:14.1130116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16901 2022-08-17T12:42:15.1779972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:15.1780517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:15.1783959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:15.1784594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:15.1898737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:15.1899373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:15.1902568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:15.1903250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:15.2509356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:15.2509925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:15.2512673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:15.2513275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:15.2543546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:15.2544269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:15.2548393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:15.2549003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:15.3362994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjas7kt8 2022-08-17T12:42:15.3363808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjas7kt8/_remote_module_non_scriptable.py 2022-08-17T12:42:15.3495177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8wxywmz 2022-08-17T12:42:15.3497431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8wxywmz/_remote_module_non_scriptable.py 2022-08-17T12:42:15.4098794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfp0yvdi 2022-08-17T12:42:15.4099553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfp0yvdi/_remote_module_non_scriptable.py 2022-08-17T12:42:15.4121027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcr02w8r 2022-08-17T12:42:15.4122475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcr02w8r/_remote_module_non_scriptable.py 2022-08-17T12:42:15.5979998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:15.6325075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:15.6740115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:15.6776765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:17.3207696Z ok (3.481s) 2022-08-17T12:42:17.3207920Z 2022-08-17T12:42:17.3208534Z ---------------------------------------------------------------------- 2022-08-17T12:42:17.3209045Z Ran 1 test in 3.481s 2022-08-17T12:42:17.3209211Z 2022-08-17T12:42:17.3209279Z OK 2022-08-17T12:42:17.3209375Z 2022-08-17T12:42:17.3209471Z Generating XML reports... 2022-08-17T12:42:17.3241563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124213.xml 2022-08-17T12:42:18.6162365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:18.6162787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:18.6164909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:18.6165274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:18.7728004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocrd9u_h 2022-08-17T12:42:18.7730472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocrd9u_h/_remote_module_non_scriptable.py 2022-08-17T12:42:19.0398219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:19.0409267Z 2022-08-17T12:42:19.0409381Z Running tests... 2022-08-17T12:42:19.0409996Z ---------------------------------------------------------------------- 2022-08-17T12:42:19.3017422Z test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17141 2022-08-17T12:42:19.3045050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17142 2022-08-17T12:42:19.3073919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17143 2022-08-17T12:42:19.3103810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17144 2022-08-17T12:42:20.3801880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:20.3802559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:20.3804915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:20.3805576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:20.3898021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:20.3898689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:20.3901789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:20.3902390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:20.4676849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:20.4677500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:20.4680704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:20.4681346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:20.4702067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:20.4702666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:20.4706027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:20.4706727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:20.5404337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5380rrr 2022-08-17T12:42:20.5405143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5380rrr/_remote_module_non_scriptable.py 2022-08-17T12:42:20.5483193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5p4icgw 2022-08-17T12:42:20.5484579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5p4icgw/_remote_module_non_scriptable.py 2022-08-17T12:42:20.6247457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1keoppy 2022-08-17T12:42:20.6248296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1keoppy/_remote_module_non_scriptable.py 2022-08-17T12:42:20.6294225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp634qv1uy 2022-08-17T12:42:20.6295615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp634qv1uy/_remote_module_non_scriptable.py 2022-08-17T12:42:20.8017196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:20.8106660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:20.8850484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:20.8926317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:21.7170251Z ok (2.676s) 2022-08-17T12:42:21.7170521Z 2022-08-17T12:42:21.7171064Z ---------------------------------------------------------------------- 2022-08-17T12:42:21.7171414Z Ran 1 test in 2.676s 2022-08-17T12:42:21.7171539Z 2022-08-17T12:42:21.7171588Z OK 2022-08-17T12:42:21.7171684Z 2022-08-17T12:42:21.7172037Z Generating XML reports... 2022-08-17T12:42:21.7203220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124219.xml 2022-08-17T12:42:22.9966526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:22.9966937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:22.9968908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:22.9969403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:23.1478591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppssgco6b 2022-08-17T12:42:23.1481920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppssgco6b/_remote_module_non_scriptable.py 2022-08-17T12:42:23.4121336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:23.4132075Z 2022-08-17T12:42:23.4132327Z Running tests... 2022-08-17T12:42:23.4132958Z ---------------------------------------------------------------------- 2022-08-17T12:42:23.6747703Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17384 2022-08-17T12:42:23.6775704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17385 2022-08-17T12:42:23.6804398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17386 2022-08-17T12:42:23.6836818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17387 2022-08-17T12:42:24.6615515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:24.6616187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:24.6619107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:24.6620051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:24.6994067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:24.6994730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:24.6997960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:24.6998639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:24.7113623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:24.7114188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:24.7117490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:24.7118026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:24.7169325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:24.7169948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:24.7174475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:24.7175105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:24.8183816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0y_9nna 2022-08-17T12:42:24.8184578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0y_9nna/_remote_module_non_scriptable.py 2022-08-17T12:42:24.8596498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg160o13 2022-08-17T12:42:24.8597586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg160o13/_remote_module_non_scriptable.py 2022-08-17T12:42:24.8687806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9sdhzdh_ 2022-08-17T12:42:24.8688839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9sdhzdh_/_remote_module_non_scriptable.py 2022-08-17T12:42:24.8754138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa7boh14 2022-08-17T12:42:24.8755363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa7boh14/_remote_module_non_scriptable.py 2022-08-17T12:42:25.0803331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:25.1212272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:25.1361036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:25.1494542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:26.0901150Z ok (2.677s) 2022-08-17T12:42:26.0901437Z 2022-08-17T12:42:26.0901866Z ---------------------------------------------------------------------- 2022-08-17T12:42:26.0902247Z Ran 1 test in 2.677s 2022-08-17T12:42:26.0902372Z 2022-08-17T12:42:26.0902439Z OK 2022-08-17T12:42:26.0902538Z 2022-08-17T12:42:26.0902623Z Generating XML reports... 2022-08-17T12:42:26.0933946Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124223.xml 2022-08-17T12:42:27.3839987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:27.3840630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:27.3844383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:27.3845013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:27.5368237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qzlk17i 2022-08-17T12:42:27.5370789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qzlk17i/_remote_module_non_scriptable.py 2022-08-17T12:42:27.8023698Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:27.8034486Z 2022-08-17T12:42:27.8034587Z Running tests... 2022-08-17T12:42:27.8035124Z ---------------------------------------------------------------------- 2022-08-17T12:42:28.0703738Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17627 2022-08-17T12:42:28.0733272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17628 2022-08-17T12:42:28.0763180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17629 2022-08-17T12:42:28.0796405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17630 2022-08-17T12:42:29.1100679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:29.1101329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:29.1104644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:29.1598568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:29.1599587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:29.1600196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:29.1602957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:29.1603898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:29.1659305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:29.1659878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:29.1664189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:29.1664876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:29.1692277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:29.1692954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:29.1697932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:29.1698638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:29.2881400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2_9_ejb 2022-08-17T12:42:29.2882223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2_9_ejb/_remote_module_non_scriptable.py 2022-08-17T12:42:29.3247499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnenwnxpi 2022-08-17T12:42:29.3249994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnenwnxpi/_remote_module_non_scriptable.py 2022-08-17T12:42:29.3294918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2u7g0px 2022-08-17T12:42:29.3296549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2u7g0px/_remote_module_non_scriptable.py 2022-08-17T12:42:29.3299632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq392tph 2022-08-17T12:42:29.3302224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq392tph/_remote_module_non_scriptable.py 2022-08-17T12:42:29.5500310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:29.5903735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:29.5904407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:29.5912805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:30.3857792Z ok (2.582s) 2022-08-17T12:42:30.3858061Z 2022-08-17T12:42:30.3858569Z ---------------------------------------------------------------------- 2022-08-17T12:42:30.3859114Z Ran 1 test in 2.582s 2022-08-17T12:42:30.3859236Z 2022-08-17T12:42:30.3859303Z OK 2022-08-17T12:42:30.3859387Z 2022-08-17T12:42:30.3859486Z Generating XML reports... 2022-08-17T12:42:30.3895854Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124227.xml 2022-08-17T12:42:31.7012318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:31.7012725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:31.7015220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:31.7015601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:31.8531680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0ekr5xc 2022-08-17T12:42:31.8534993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0ekr5xc/_remote_module_non_scriptable.py 2022-08-17T12:42:32.1166917Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:32.1177452Z 2022-08-17T12:42:32.1177985Z Running tests... 2022-08-17T12:42:32.1178637Z ---------------------------------------------------------------------- 2022-08-17T12:42:32.3790887Z test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s) 2022-08-17T12:42:32.3791428Z 2022-08-17T12:42:32.3791652Z ---------------------------------------------------------------------- 2022-08-17T12:42:32.3791911Z Ran 1 test in 0.261s 2022-08-17T12:42:32.3792031Z 2022-08-17T12:42:32.3792106Z OK (skipped=1) 2022-08-17T12:42:32.3792232Z 2022-08-17T12:42:32.3792307Z Generating XML reports... 2022-08-17T12:42:32.3815937Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124232.xml 2022-08-17T12:42:33.6014199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:33.6014870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:33.6017392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:33.6017997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:33.7508321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzslcv_1 2022-08-17T12:42:33.7511687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzslcv_1/_remote_module_non_scriptable.py 2022-08-17T12:42:34.0113368Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:34.0123978Z 2022-08-17T12:42:34.0124360Z Running tests... 2022-08-17T12:42:34.0124989Z ---------------------------------------------------------------------- 2022-08-17T12:42:34.2724327Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17860 2022-08-17T12:42:34.2752623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17861 2022-08-17T12:42:34.2780512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17862 2022-08-17T12:42:34.2810924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17863 2022-08-17T12:42:35.3392502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:35.3393138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:35.3395856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:35.3396433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:35.3484777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:35.3485419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:35.3489100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:35.3489774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:35.3894067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:35.3894628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:35.3898261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:35.3898932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:35.4246327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:35.4247430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:35.4249202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:35.4249812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:35.5019309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f9hrnyk 2022-08-17T12:42:35.5022387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f9hrnyk/_remote_module_non_scriptable.py 2022-08-17T12:42:35.5098065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9loatna 2022-08-17T12:42:35.5098918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9loatna/_remote_module_non_scriptable.py 2022-08-17T12:42:35.5654473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhlh8x8p 2022-08-17T12:42:35.5655367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhlh8x8p/_remote_module_non_scriptable.py 2022-08-17T12:42:35.5839068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mxun_qq 2022-08-17T12:42:35.5840215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mxun_qq/_remote_module_non_scriptable.py 2022-08-17T12:42:35.7653187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:35.7735183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:35.8272890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:35.8446968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:36.0757765Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:36.0851693Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:42:36.0854839Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.0857952Z Traceback (most recent call last): 2022-08-17T12:42:36.0862792Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.0863649Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.0865051Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.0865782Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.0866724Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.0867388Z self.optim.step() 2022-08-17T12:42:36.0868216Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.0868816Z return func(*args, **kwargs) 2022-08-17T12:42:36.0869782Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.0870553Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.0871091Z ValueError: Error running optimizer. 2022-08-17T12:42:36.0871377Z 2022-08-17T12:42:36.0871587Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:42:36.0890976Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.0891467Z Traceback (most recent call last): 2022-08-17T12:42:36.0892206Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.0894100Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.0895101Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.0895703Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.0896435Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.0898288Z self.optim.step() 2022-08-17T12:42:36.0898957Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.0899457Z return func(*args, **kwargs) 2022-08-17T12:42:36.0900217Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.0900829Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.0901267Z ValueError: Error running optimizer. 2022-08-17T12:42:36.0901495Z 2022-08-17T12:42:36.0902927Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:36.0959725Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:36.0967558Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:36.1474657Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:42:36.1475353Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1476207Z Traceback (most recent call last): 2022-08-17T12:42:36.1476977Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1477515Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1478061Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1478587Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1479062Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1479399Z self.optim.step() 2022-08-17T12:42:36.1479955Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1480438Z return func(*args, **kwargs) 2022-08-17T12:42:36.1481160Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1481603Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1481871Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1482058Z 2022-08-17T12:42:36.1482435Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:42:36.1482883Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1483289Z Traceback (most recent call last): 2022-08-17T12:42:36.1483941Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1484292Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1484907Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1485262Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1485760Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1486340Z self.optim.step() 2022-08-17T12:42:36.1486830Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1489089Z return func(*args, **kwargs) 2022-08-17T12:42:36.1489921Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1490499Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1490917Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1491145Z 2022-08-17T12:42:36.1622358Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:42:36.1628961Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1629421Z Traceback (most recent call last): 2022-08-17T12:42:36.1630249Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1630930Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1631738Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1632277Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1632958Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1633462Z self.optim.step() 2022-08-17T12:42:36.1634165Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1634672Z return func(*args, **kwargs) 2022-08-17T12:42:36.1635492Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1636120Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1636547Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1636812Z 2022-08-17T12:42:36.1637029Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:42:36.1637470Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1637709Z Traceback (most recent call last): 2022-08-17T12:42:36.1638189Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1638547Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1639094Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1639673Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1640084Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1640445Z self.optim.step() 2022-08-17T12:42:36.1640828Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1641164Z return func(*args, **kwargs) 2022-08-17T12:42:36.1641892Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1642404Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1642765Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1642986Z 2022-08-17T12:42:36.1643142Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:42:36.1643610Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1643978Z Traceback (most recent call last): 2022-08-17T12:42:36.1644620Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1645151Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1646133Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1646807Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1647415Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1647814Z self.optim.step() 2022-08-17T12:42:36.1648309Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1648749Z return func(*args, **kwargs) 2022-08-17T12:42:36.1649199Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1649558Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1649807Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1649953Z 2022-08-17T12:42:36.1650050Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:42:36.1650326Z ValueError('Error running optimizer.') 2022-08-17T12:42:36.1650550Z Traceback (most recent call last): 2022-08-17T12:42:36.1650965Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:36.1651322Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:36.1651796Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 97, in _local_optimizer_step 2022-08-17T12:42:36.1652110Z local_optim.step(autograd_ctx_id) 2022-08-17T12:42:36.1652513Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 87, in step 2022-08-17T12:42:36.1652805Z self.optim.step() 2022-08-17T12:42:36.1653161Z File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 127, in wrapper 2022-08-17T12:42:36.1653455Z return func(*args, **kwargs) 2022-08-17T12:42:36.1654019Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step 2022-08-17T12:42:36.1654545Z raise ValueError("Error running optimizer.") 2022-08-17T12:42:36.1654953Z ValueError: Error running optimizer. 2022-08-17T12:42:36.1655186Z 2022-08-17T12:42:36.5874078Z ok (2.575s) 2022-08-17T12:42:36.5874585Z 2022-08-17T12:42:36.5875194Z ---------------------------------------------------------------------- 2022-08-17T12:42:36.5875678Z Ran 1 test in 2.575s 2022-08-17T12:42:36.5875883Z 2022-08-17T12:42:36.5875938Z OK 2022-08-17T12:42:36.5876034Z 2022-08-17T12:42:36.5876128Z Generating XML reports... 2022-08-17T12:42:36.5908296Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124234.xml 2022-08-17T12:42:37.8750969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:37.8751604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:37.8753877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:37.8754394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:38.0277094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_65yns2 2022-08-17T12:42:38.0280445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_65yns2/_remote_module_non_scriptable.py 2022-08-17T12:42:38.2900149Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:38.2910840Z 2022-08-17T12:42:38.2910996Z Running tests... 2022-08-17T12:42:38.2911641Z ---------------------------------------------------------------------- 2022-08-17T12:42:38.5520155Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18103 2022-08-17T12:42:38.5556342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18104 2022-08-17T12:42:38.5583195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18105 2022-08-17T12:42:38.5612642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18106 2022-08-17T12:42:39.5558467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:39.5559174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:39.5562774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:39.5563428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:39.5646848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:39.5647939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:39.5651465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:39.5652112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:39.5677545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:39.5678164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:39.5682878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:39.5683415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:39.6090055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:39.6090758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:39.6094654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:39.6095409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:39.7197682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4baqsn9 2022-08-17T12:42:39.7198413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4baqsn9/_remote_module_non_scriptable.py 2022-08-17T12:42:39.7278451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yl638qj 2022-08-17T12:42:39.7279323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yl638qj/_remote_module_non_scriptable.py 2022-08-17T12:42:39.7289856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vtvtyjp 2022-08-17T12:42:39.7291975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vtvtyjp/_remote_module_non_scriptable.py 2022-08-17T12:42:39.7691544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsrt2vwt 2022-08-17T12:42:39.7692986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsrt2vwt/_remote_module_non_scriptable.py 2022-08-17T12:42:39.9812888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:39.9874045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:39.9918632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:40.0351719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:40.2525387Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:40.2537415Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:42:40.2539302Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.2543579Z Traceback (most recent call last): 2022-08-17T12:42:40.2545135Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.2549681Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.2551348Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.2555685Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.2557260Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.2561428Z **kwargs) 2022-08-17T12:42:40.2562602Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.2566541Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.2567549Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.2567885Z 2022-08-17T12:42:40.2568095Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:42:40.2571566Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.2573213Z Traceback (most recent call last): 2022-08-17T12:42:40.2575436Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.2577314Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.2579456Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.2581361Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.2583437Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.2585115Z **kwargs) 2022-08-17T12:42:40.2587257Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.2589342Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.2590991Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.2592403Z 2022-08-17T12:42:40.2719838Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:40.2725596Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:40.2729927Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 2022-08-17T12:42:40.2806663Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:42:40.2807063Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.2807306Z Traceback (most recent call last): 2022-08-17T12:42:40.2807737Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.2808290Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.2808847Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.2809401Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.2810055Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.2810464Z **kwargs) 2022-08-17T12:42:40.2811106Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.2811640Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.2812024Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.2812251Z 2022-08-17T12:42:40.2814712Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:42:40.2815215Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.2815617Z Traceback (most recent call last): 2022-08-17T12:42:40.2816264Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.2816630Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.2817089Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.2817435Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.2817866Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.2818142Z **kwargs) 2022-08-17T12:42:40.2818554Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.2818897Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.2819262Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.2819403Z 2022-08-17T12:42:40.2900656Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:42:40.2901213Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.2901675Z Traceback (most recent call last): 2022-08-17T12:42:40.2902146Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.2902503Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.2902950Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.2903311Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.2903744Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.2904214Z **kwargs) 2022-08-17T12:42:40.2904633Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.2904994Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.2905245Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.2905372Z 2022-08-17T12:42:40.3263661Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:42:40.3264269Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.3264702Z Traceback (most recent call last): 2022-08-17T12:42:40.3265407Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.3266052Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.3266974Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.3267608Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.3268395Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.3268894Z **kwargs) 2022-08-17T12:42:40.3269846Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.3270207Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.3270497Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.3270638Z 2022-08-17T12:42:40.3304221Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:42:40.3304823Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.3305229Z Traceback (most recent call last): 2022-08-17T12:42:40.3305990Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.3306666Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.3307466Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.3308052Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.3308803Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.3309274Z **kwargs) 2022-08-17T12:42:40.3309950Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.3310535Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.3310977Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.3311224Z 2022-08-17T12:42:40.3340208Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:42:40.3340911Z ValueError('Error creating optimizer.') 2022-08-17T12:42:40.3341429Z Traceback (most recent call last): 2022-08-17T12:42:40.3342394Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:42:40.3343172Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:42:40.3344106Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 92, in _new_local_optimizer 2022-08-17T12:42:40.3344859Z _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs)) 2022-08-17T12:42:40.3345844Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 79, in __init__ 2022-08-17T12:42:40.3346444Z **kwargs) 2022-08-17T12:42:40.3347375Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__ 2022-08-17T12:42:40.3348147Z raise ValueError("Error creating optimizer.") 2022-08-17T12:42:40.3348672Z ValueError: Error creating optimizer. 2022-08-17T12:42:40.3348980Z 2022-08-17T12:42:40.7672710Z ok (2.476s) 2022-08-17T12:42:40.7673197Z 2022-08-17T12:42:40.7673682Z ---------------------------------------------------------------------- 2022-08-17T12:42:40.7673958Z Ran 1 test in 2.476s 2022-08-17T12:42:40.7674080Z 2022-08-17T12:42:40.7674144Z OK 2022-08-17T12:42:40.7674241Z 2022-08-17T12:42:40.7674336Z Generating XML reports... 2022-08-17T12:42:40.7709652Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124238.xml 2022-08-17T12:42:42.0820586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:42.0820977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:42.0823552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:42.0823936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:42.2365185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kkt3plr 2022-08-17T12:42:42.2368985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kkt3plr/_remote_module_non_scriptable.py 2022-08-17T12:42:42.5033270Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:42.5043885Z 2022-08-17T12:42:42.5044095Z Running tests... 2022-08-17T12:42:42.5044683Z ---------------------------------------------------------------------- 2022-08-17T12:42:42.7640049Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s) 2022-08-17T12:42:42.7640590Z 2022-08-17T12:42:42.7640794Z ---------------------------------------------------------------------- 2022-08-17T12:42:42.7641087Z Ran 1 test in 0.259s 2022-08-17T12:42:42.7641207Z 2022-08-17T12:42:42.7641285Z OK (skipped=1) 2022-08-17T12:42:42.7641400Z 2022-08-17T12:42:42.7641490Z Generating XML reports... 2022-08-17T12:42:42.7668150Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124242.xml 2022-08-17T12:42:44.0064084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:44.0064476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:44.0067259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:44.0067632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:44.1583718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89oot8xd 2022-08-17T12:42:44.1586829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89oot8xd/_remote_module_non_scriptable.py 2022-08-17T12:42:44.4246176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:44.4256794Z 2022-08-17T12:42:44.4256899Z Running tests... 2022-08-17T12:42:44.4257370Z ---------------------------------------------------------------------- 2022-08-17T12:42:44.6875773Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18336 2022-08-17T12:42:44.6904726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18337 2022-08-17T12:42:44.6933625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18338 2022-08-17T12:42:44.6966084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18339 2022-08-17T12:42:45.7382031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:45.7383024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:45.7384661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:45.7385279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:45.7493006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:45.7496428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:45.7497317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:45.7497971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:45.7599173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:45.7599899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:45.7603862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:45.7604530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:45.7610067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:45.7610693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:45.7613084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:45.7613739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:45.9021397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mcjfi02 2022-08-17T12:42:45.9022776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mcjfi02/_remote_module_non_scriptable.py 2022-08-17T12:42:45.9107950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s_h8rmv 2022-08-17T12:42:45.9109274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s_h8rmv/_remote_module_non_scriptable.py 2022-08-17T12:42:45.9226891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx47ysi2f 2022-08-17T12:42:45.9228832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx47ysi2f/_remote_module_non_scriptable.py 2022-08-17T12:42:45.9406193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpv5yng5 2022-08-17T12:42:45.9407372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpv5yng5/_remote_module_non_scriptable.py 2022-08-17T12:42:46.1585099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:46.1703385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:46.1824448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:46.1995445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:46.9026871Z ok (2.477s) 2022-08-17T12:42:46.9027145Z 2022-08-17T12:42:46.9027696Z ---------------------------------------------------------------------- 2022-08-17T12:42:46.9027999Z Ran 1 test in 2.477s 2022-08-17T12:42:46.9028120Z 2022-08-17T12:42:46.9028171Z OK 2022-08-17T12:42:46.9028267Z 2022-08-17T12:42:46.9028363Z Generating XML reports... 2022-08-17T12:42:46.9061898Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124244.xml 2022-08-17T12:42:48.1973253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:48.1973645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:48.1976180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:48.1976627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:48.3461402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wpxegil 2022-08-17T12:42:48.3464895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wpxegil/_remote_module_non_scriptable.py 2022-08-17T12:42:48.6055437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:48.6066677Z 2022-08-17T12:42:48.6067007Z Running tests... 2022-08-17T12:42:48.6067657Z ---------------------------------------------------------------------- 2022-08-17T12:42:48.8697096Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18569 2022-08-17T12:42:48.8725691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18570 2022-08-17T12:42:48.8754222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18571 2022-08-17T12:42:48.8785573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18572 2022-08-17T12:42:49.9293971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:49.9294674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:49.9296903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:49.9297580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:49.9663445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:49.9664335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:49.9666045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:49.9666696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:49.9779070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:49.9779709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:49.9782773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:49.9783342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:50.0180287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:50.0180981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:50.0183583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:50.0184310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:50.1088963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq68z02x 2022-08-17T12:42:50.1089796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq68z02x/_remote_module_non_scriptable.py 2022-08-17T12:42:50.1316120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r0pstpp 2022-08-17T12:42:50.1317221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r0pstpp/_remote_module_non_scriptable.py 2022-08-17T12:42:50.1413527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fd5euz1 2022-08-17T12:42:50.1414791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fd5euz1/_remote_module_non_scriptable.py 2022-08-17T12:42:50.1773601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsp9pgvv 2022-08-17T12:42:50.1774605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsp9pgvv/_remote_module_non_scriptable.py 2022-08-17T12:42:50.3730616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:50.3960747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:50.4062862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:50.4411364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:51.1848875Z ok (2.578s) 2022-08-17T12:42:51.1849139Z 2022-08-17T12:42:51.1849696Z ---------------------------------------------------------------------- 2022-08-17T12:42:51.1849960Z Ran 1 test in 2.578s 2022-08-17T12:42:51.1850084Z 2022-08-17T12:42:51.1850150Z OK 2022-08-17T12:42:51.1850247Z 2022-08-17T12:42:51.1850344Z Generating XML reports... 2022-08-17T12:42:51.1882366Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124248.xml 2022-08-17T12:42:52.5053148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:52.5054119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:52.5056511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:52.5057098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:52.6583115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphig37ff6 2022-08-17T12:42:52.6586742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphig37ff6/_remote_module_non_scriptable.py 2022-08-17T12:42:52.9199943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:52.9211191Z 2022-08-17T12:42:52.9211507Z Running tests... 2022-08-17T12:42:52.9212201Z ---------------------------------------------------------------------- 2022-08-17T12:42:53.1820327Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18812 2022-08-17T12:42:53.1848420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18813 2022-08-17T12:42:53.1876484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18814 2022-08-17T12:42:53.1906689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18815 2022-08-17T12:42:54.2796803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:54.2797436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:54.2800165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:54.2800778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:54.2865237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:54.2865891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:54.2869166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:54.2869839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:54.3470191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:54.3470854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:54.3473181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:54.3474196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:54.4008582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:54.4009262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:54.4011518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:54.4012198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:54.4380023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuraw3ee0 2022-08-17T12:42:54.4380790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuraw3ee0/_remote_module_non_scriptable.py 2022-08-17T12:42:54.4455957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfyf08c5 2022-08-17T12:42:54.4457452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfyf08c5/_remote_module_non_scriptable.py 2022-08-17T12:42:54.5054995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu_dv6au 2022-08-17T12:42:54.5056132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu_dv6au/_remote_module_non_scriptable.py 2022-08-17T12:42:54.5594213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80tx3ut0 2022-08-17T12:42:54.5595041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80tx3ut0/_remote_module_non_scriptable.py 2022-08-17T12:42:54.6995381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:54.7090928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:54.7683400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:54.8231656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:55.5970868Z ok (2.676s) 2022-08-17T12:42:55.5971142Z 2022-08-17T12:42:55.5971697Z ---------------------------------------------------------------------- 2022-08-17T12:42:55.5972108Z Ran 1 test in 2.676s 2022-08-17T12:42:55.5972231Z 2022-08-17T12:42:55.5972299Z OK 2022-08-17T12:42:55.5972398Z 2022-08-17T12:42:55.5972508Z Generating XML reports... 2022-08-17T12:42:55.6005232Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124252.xml 2022-08-17T12:42:56.8995846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:56.8996283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:56.8998928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:56.8999338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:57.0521366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmduhm51k 2022-08-17T12:42:57.0524811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmduhm51k/_remote_module_non_scriptable.py 2022-08-17T12:42:57.3192420Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:42:57.3204022Z 2022-08-17T12:42:57.3204468Z Running tests... 2022-08-17T12:42:57.3204906Z ---------------------------------------------------------------------- 2022-08-17T12:42:57.5844695Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19055 2022-08-17T12:42:57.5873827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19056 2022-08-17T12:42:57.5903812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19057 2022-08-17T12:42:57.5935434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19058 2022-08-17T12:42:58.6132103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:58.6132790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:58.6136642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:58.6137331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:58.6402149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:58.6402779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:58.6405578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:58.6406446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:58.6496473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:58.6497147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:58.6501656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:58.6502322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:58.6674197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:42:58.6674866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:42:58.6676901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:42:58.6677498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:42:58.7711762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghl_qoaz 2022-08-17T12:42:58.7712978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghl_qoaz/_remote_module_non_scriptable.py 2022-08-17T12:42:58.8017697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2624oi4v 2022-08-17T12:42:58.8018529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2624oi4v/_remote_module_non_scriptable.py 2022-08-17T12:42:58.8095788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4xu7hv5 2022-08-17T12:42:58.8097980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4xu7hv5/_remote_module_non_scriptable.py 2022-08-17T12:42:58.8253166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43h1mgrr 2022-08-17T12:42:58.8254215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43h1mgrr/_remote_module_non_scriptable.py 2022-08-17T12:42:59.0347848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:42:59.0673407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:42:59.0721943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:42:59.0850023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:42:59.8997328Z ok (2.579s) 2022-08-17T12:42:59.8997603Z 2022-08-17T12:42:59.8998120Z ---------------------------------------------------------------------- 2022-08-17T12:42:59.8998374Z Ran 1 test in 2.579s 2022-08-17T12:42:59.8998498Z 2022-08-17T12:42:59.8998563Z OK 2022-08-17T12:42:59.8998660Z 2022-08-17T12:42:59.8998759Z Generating XML reports... 2022-08-17T12:42:59.9031510Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124257.xml 2022-08-17T12:43:01.1821457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:01.1821926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:01.1823849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:01.1824340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:01.3333062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy5xkf3c 2022-08-17T12:43:01.3336405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy5xkf3c/_remote_module_non_scriptable.py 2022-08-17T12:43:01.6008735Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:01.6018680Z 2022-08-17T12:43:01.6018831Z Running tests... 2022-08-17T12:43:01.6019497Z ---------------------------------------------------------------------- 2022-08-17T12:43:01.8681887Z test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19288 2022-08-17T12:43:01.8710416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19289 2022-08-17T12:43:01.8740217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19290 2022-08-17T12:43:01.8773500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19291 2022-08-17T12:43:02.8931312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:02.8931936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:02.8934633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:02.8935267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:02.9030998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:02.9031610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:02.9035158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:02.9035816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:02.9142915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:02.9143624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:02.9146576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:02.9147319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:02.9427394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:02.9428052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:02.9431419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:02.9432004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:03.0532731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cpwyle7 2022-08-17T12:43:03.0533572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cpwyle7/_remote_module_non_scriptable.py 2022-08-17T12:43:03.0620212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4v8gts6 2022-08-17T12:43:03.0621444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4v8gts6/_remote_module_non_scriptable.py 2022-08-17T12:43:03.0748077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w65ivv7 2022-08-17T12:43:03.0749169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w65ivv7/_remote_module_non_scriptable.py 2022-08-17T12:43:03.1024437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2vasp2p 2022-08-17T12:43:03.1025311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2vasp2p/_remote_module_non_scriptable.py 2022-08-17T12:43:03.3141338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:03.3237064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:03.3377612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:03.3664633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:04.0832689Z ok (2.481s) 2022-08-17T12:43:04.0833325Z 2022-08-17T12:43:04.0834124Z ---------------------------------------------------------------------- 2022-08-17T12:43:04.0834503Z Ran 1 test in 2.481s 2022-08-17T12:43:04.0834667Z 2022-08-17T12:43:04.0834735Z OK 2022-08-17T12:43:04.0834843Z 2022-08-17T12:43:04.0834940Z Generating XML reports... 2022-08-17T12:43:04.0868400Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124301.xml 2022-08-17T12:43:05.3872905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:05.3873306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:05.3875363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:05.3875757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:05.5398428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6a01voko 2022-08-17T12:43:05.5402101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6a01voko/_remote_module_non_scriptable.py 2022-08-17T12:43:05.8106099Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:05.8117316Z 2022-08-17T12:43:05.8117432Z Running tests... 2022-08-17T12:43:05.8118070Z ---------------------------------------------------------------------- 2022-08-17T12:43:06.0803360Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19527 2022-08-17T12:43:06.0833596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19528 2022-08-17T12:43:06.0862211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19529 2022-08-17T12:43:06.0892752Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19530 2022-08-17T12:43:07.0975808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:07.0976434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:07.0978993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:07.0979612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:07.1264175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:07.1264846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:07.1266600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:07.1267249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:07.1523193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:07.1524005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:07.1526943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:07.1527530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:07.1634670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:07.1635366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:07.1638812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:07.1639481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:07.2549071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby68r2gf 2022-08-17T12:43:07.2550179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby68r2gf/_remote_module_non_scriptable.py 2022-08-17T12:43:07.2850310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubfspv5m 2022-08-17T12:43:07.2851134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubfspv5m/_remote_module_non_scriptable.py 2022-08-17T12:43:07.3144816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8b_wd0s2 2022-08-17T12:43:07.3145659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8b_wd0s2/_remote_module_non_scriptable.py 2022-08-17T12:43:07.3228099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql3qxtol 2022-08-17T12:43:07.3229272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql3qxtol/_remote_module_non_scriptable.py 2022-08-17T12:43:07.5183437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:07.5482557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:07.5837968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:07.5843510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:08.3956989Z ok (2.584s) 2022-08-17T12:43:08.3957162Z 2022-08-17T12:43:08.3957637Z ---------------------------------------------------------------------- 2022-08-17T12:43:08.3957894Z Ran 1 test in 2.584s 2022-08-17T12:43:08.3958019Z 2022-08-17T12:43:08.3958084Z OK 2022-08-17T12:43:08.3958184Z 2022-08-17T12:43:08.3958282Z Generating XML reports... 2022-08-17T12:43:08.3994290Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124305.xml 2022-08-17T12:43:09.6930616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:09.6931318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:09.6934277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:09.6934875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:09.8438859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv77cvr5d 2022-08-17T12:43:09.8442178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv77cvr5d/_remote_module_non_scriptable.py 2022-08-17T12:43:10.1066354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:10.1077401Z 2022-08-17T12:43:10.1077830Z Running tests... 2022-08-17T12:43:10.1078532Z ---------------------------------------------------------------------- 2022-08-17T12:43:10.3709211Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19754 2022-08-17T12:43:10.3738474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19755 2022-08-17T12:43:10.3767101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19756 2022-08-17T12:43:10.3798457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19757 2022-08-17T12:43:11.3683477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:11.3684120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:11.3685651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:11.3686493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:11.3746002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:11.3746723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:11.3750382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:11.3751040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:11.3888924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:11.3889569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:11.3890857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:11.3891506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:11.4248847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:11.4249463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:11.4250438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:11.4251039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:11.5290799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp3hvaq9 2022-08-17T12:43:11.5291778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp3hvaq9/_remote_module_non_scriptable.py 2022-08-17T12:43:11.5356476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixjlrpan 2022-08-17T12:43:11.5357982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixjlrpan/_remote_module_non_scriptable.py 2022-08-17T12:43:11.5499660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3ron1n5 2022-08-17T12:43:11.5500904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3ron1n5/_remote_module_non_scriptable.py 2022-08-17T12:43:11.5833015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7xm1x1e 2022-08-17T12:43:11.5833889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7xm1x1e/_remote_module_non_scriptable.py 2022-08-17T12:43:11.7909550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:11.7960624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:11.8116400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:11.8431556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:12.5858814Z ok (2.478s) 2022-08-17T12:43:12.5859059Z 2022-08-17T12:43:12.5859602Z ---------------------------------------------------------------------- 2022-08-17T12:43:12.5860065Z Ran 1 test in 2.478s 2022-08-17T12:43:12.5860198Z 2022-08-17T12:43:12.5860265Z OK 2022-08-17T12:43:12.5860347Z 2022-08-17T12:43:12.5860440Z Generating XML reports... 2022-08-17T12:43:12.5893711Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124310.xml 2022-08-17T12:43:13.9119476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:13.9120174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:13.9123377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:13.9123859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:14.0643938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqwm_qsj 2022-08-17T12:43:14.0646861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqwm_qsj/_remote_module_non_scriptable.py 2022-08-17T12:43:14.3305707Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:14.3316449Z 2022-08-17T12:43:14.3316552Z Running tests... 2022-08-17T12:43:14.3317398Z ---------------------------------------------------------------------- 2022-08-17T12:43:14.5920688Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19981 2022-08-17T12:43:14.5950120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19982 2022-08-17T12:43:14.5978842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19983 2022-08-17T12:43:14.6010054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19984 2022-08-17T12:43:15.6368322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:15.6368994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:15.6372161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:15.6372855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:15.6464537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:15.6465239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:15.6468684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:15.6469337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:15.6538677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:15.6539321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:15.6542688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:15.6543357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:15.6641823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:15.6645656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:15.6646769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:15.6647415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:15.7966949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkf5juzb 2022-08-17T12:43:15.7968378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkf5juzb/_remote_module_non_scriptable.py 2022-08-17T12:43:15.8054180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf0u_3ba 2022-08-17T12:43:15.8056273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf0u_3ba/_remote_module_non_scriptable.py 2022-08-17T12:43:15.8126886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t7bbclz 2022-08-17T12:43:15.8128275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t7bbclz/_remote_module_non_scriptable.py 2022-08-17T12:43:15.8242510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx78yiww 2022-08-17T12:43:15.8243758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx78yiww/_remote_module_non_scriptable.py 2022-08-17T12:43:16.0585895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:16.0643654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:16.0697036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:16.0857383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:16.8070530Z ok (2.475s) 2022-08-17T12:43:16.8070699Z 2022-08-17T12:43:16.8071193Z ---------------------------------------------------------------------- 2022-08-17T12:43:16.8071706Z Ran 1 test in 2.475s 2022-08-17T12:43:16.8071851Z 2022-08-17T12:43:16.8071915Z OK 2022-08-17T12:43:16.8072012Z 2022-08-17T12:43:16.8072358Z Generating XML reports... 2022-08-17T12:43:16.8105896Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124314.xml 2022-08-17T12:43:18.0880467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:18.0880865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:18.0883368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:18.0883736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:18.2448334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmii7tidy 2022-08-17T12:43:18.2451927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmii7tidy/_remote_module_non_scriptable.py 2022-08-17T12:43:18.5066699Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:18.5077074Z 2022-08-17T12:43:18.5077186Z Running tests... 2022-08-17T12:43:18.5077816Z ---------------------------------------------------------------------- 2022-08-17T12:43:18.7653944Z test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20208 2022-08-17T12:43:18.7682354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20209 2022-08-17T12:43:18.7710143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20210 2022-08-17T12:43:18.7740976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20211 2022-08-17T12:43:19.7809407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:19.7810055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:19.7812831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:19.7813484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:19.7952975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:19.7953613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:19.7956568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:19.7957183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:19.8137209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:19.8137882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:19.8141944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:19.8142626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:19.8710763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:19.8711526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:19.8713960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:19.8714681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:19.9396798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig18pspv 2022-08-17T12:43:19.9398018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig18pspv/_remote_module_non_scriptable.py 2022-08-17T12:43:19.9534728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c2sp6vf 2022-08-17T12:43:19.9536020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c2sp6vf/_remote_module_non_scriptable.py 2022-08-17T12:43:19.9733937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mz_s_b4 2022-08-17T12:43:19.9734958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mz_s_b4/_remote_module_non_scriptable.py 2022-08-17T12:43:20.0282794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvhafs7r 2022-08-17T12:43:20.0284031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvhafs7r/_remote_module_non_scriptable.py 2022-08-17T12:43:20.1998891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:20.2121577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:20.2382716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:20.2898640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:20.9801259Z ok (2.472s) 2022-08-17T12:43:20.9801491Z 2022-08-17T12:43:20.9801932Z ---------------------------------------------------------------------- 2022-08-17T12:43:20.9802337Z Ran 1 test in 2.472s 2022-08-17T12:43:20.9802498Z 2022-08-17T12:43:20.9802590Z OK 2022-08-17T12:43:20.9802737Z 2022-08-17T12:43:20.9802856Z Generating XML reports... 2022-08-17T12:43:20.9838789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124318.xml 2022-08-17T12:43:22.2745610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:22.2745999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:22.2748581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:22.2748970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:22.4310022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj8fkyar 2022-08-17T12:43:22.4313459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj8fkyar/_remote_module_non_scriptable.py 2022-08-17T12:43:22.6944880Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:22.6956368Z 2022-08-17T12:43:22.6956677Z Running tests... 2022-08-17T12:43:22.6957345Z ---------------------------------------------------------------------- 2022-08-17T12:43:22.9589789Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20431 2022-08-17T12:43:22.9618501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20432 2022-08-17T12:43:22.9648090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20433 2022-08-17T12:43:22.9679072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20434 2022-08-17T12:43:24.0186504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:24.0187142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:24.0190045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:24.0190717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:24.0405606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:24.0406524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:24.0409275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:24.0410161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:24.0777060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:24.0777750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:24.0780759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:24.0781456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:24.0793334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:24.0793928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:24.0798580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:24.0799228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:24.1788642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoc0u19zs 2022-08-17T12:43:24.1791602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoc0u19zs/_remote_module_non_scriptable.py 2022-08-17T12:43:24.1996433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0z6nrr0 2022-08-17T12:43:24.1997186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0z6nrr0/_remote_module_non_scriptable.py 2022-08-17T12:43:24.2389504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhyaxbm5 2022-08-17T12:43:24.2390463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhyaxbm5/_remote_module_non_scriptable.py 2022-08-17T12:43:24.2407171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dilbcod 2022-08-17T12:43:24.2409697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dilbcod/_remote_module_non_scriptable.py 2022-08-17T12:43:24.4431088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:24.4642407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:24.5027473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:24.5071161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:25.2742421Z ok (2.578s) 2022-08-17T12:43:25.2742672Z 2022-08-17T12:43:25.2743048Z ---------------------------------------------------------------------- 2022-08-17T12:43:25.2743448Z Ran 1 test in 2.579s 2022-08-17T12:43:25.2743836Z 2022-08-17T12:43:25.2743902Z OK 2022-08-17T12:43:25.2744002Z 2022-08-17T12:43:25.2744099Z Generating XML reports... 2022-08-17T12:43:25.2776128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124322.xml 2022-08-17T12:43:26.5888610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:26.5889274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:26.5892289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:26.5892902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:26.7414648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci_tg5vi 2022-08-17T12:43:26.7418590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci_tg5vi/_remote_module_non_scriptable.py 2022-08-17T12:43:27.0067757Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:27.0079097Z 2022-08-17T12:43:27.0079529Z Running tests... 2022-08-17T12:43:27.0080195Z ---------------------------------------------------------------------- 2022-08-17T12:43:27.2758440Z test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20654 2022-08-17T12:43:27.2789524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20655 2022-08-17T12:43:27.2819857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20656 2022-08-17T12:43:27.2852133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20657 2022-08-17T12:43:28.3075721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:28.3076370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:28.3077614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:28.3078244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:28.3565784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:28.3566645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:28.3569194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:28.3569842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:28.3606174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:28.3606808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:28.3610211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:28.3610899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:28.3705099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:28.3705755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:28.3709624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:28.3710193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:28.4667568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfd9c9awa 2022-08-17T12:43:28.4668543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfd9c9awa/_remote_module_non_scriptable.py 2022-08-17T12:43:28.5151313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdup8dit 2022-08-17T12:43:28.5152181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdup8dit/_remote_module_non_scriptable.py 2022-08-17T12:43:28.5190208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetsj9wju 2022-08-17T12:43:28.5191943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetsj9wju/_remote_module_non_scriptable.py 2022-08-17T12:43:28.5290915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynti88la 2022-08-17T12:43:28.5292284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynti88la/_remote_module_non_scriptable.py 2022-08-17T12:43:28.7530449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:28.7780244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:28.7784995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:28.7940821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:29.4912298Z ok (2.483s) 2022-08-17T12:43:29.4912537Z 2022-08-17T12:43:29.4912980Z ---------------------------------------------------------------------- 2022-08-17T12:43:29.4913703Z Ran 1 test in 2.483s 2022-08-17T12:43:29.4913915Z 2022-08-17T12:43:29.4914016Z OK 2022-08-17T12:43:29.4914172Z 2022-08-17T12:43:29.4914298Z Generating XML reports... 2022-08-17T12:43:29.4949461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124327.xml 2022-08-17T12:43:30.7934005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:30.7934392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:30.7937083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:30.7937477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:30.9442998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfft83r1h 2022-08-17T12:43:30.9446570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfft83r1h/_remote_module_non_scriptable.py 2022-08-17T12:43:31.2050604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:31.2061318Z 2022-08-17T12:43:31.2061487Z Running tests... 2022-08-17T12:43:31.2061942Z ---------------------------------------------------------------------- 2022-08-17T12:43:31.4656782Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20877 2022-08-17T12:43:31.4684128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20878 2022-08-17T12:43:31.4712849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20879 2022-08-17T12:43:31.4744985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20880 2022-08-17T12:43:32.4834964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:32.4835629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:32.4838613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:32.4839274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:32.5173294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:32.5173932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:32.5177460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:32.5178466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:32.5334486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:32.5335040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:32.5338108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:32.5338800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:32.5379929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:32.5380499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:32.5385066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:32.5385701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:32.6421147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rj6l1ef 2022-08-17T12:43:32.6422488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rj6l1ef/_remote_module_non_scriptable.py 2022-08-17T12:43:32.6786962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t_pifu4 2022-08-17T12:43:32.6787951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t_pifu4/_remote_module_non_scriptable.py 2022-08-17T12:43:32.6928933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxz9kvi9 2022-08-17T12:43:32.6929979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxz9kvi9/_remote_module_non_scriptable.py 2022-08-17T12:43:32.6981827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnc5ge84l 2022-08-17T12:43:32.6983145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnc5ge84l/_remote_module_non_scriptable.py 2022-08-17T12:43:32.9061358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:32.9434793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:32.9631331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:32.9776141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:33.5803506Z ok (2.374s) 2022-08-17T12:43:33.5803766Z 2022-08-17T12:43:33.5804304Z ---------------------------------------------------------------------- 2022-08-17T12:43:33.5804677Z Ran 1 test in 2.374s 2022-08-17T12:43:33.5804801Z 2022-08-17T12:43:33.5804864Z OK 2022-08-17T12:43:33.5804960Z 2022-08-17T12:43:33.5805055Z Generating XML reports... 2022-08-17T12:43:33.5837809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124331.xml 2022-08-17T12:43:34.8767275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:34.8767682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:34.8770524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:34.8771023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:35.0285295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zh4ja1o 2022-08-17T12:43:35.0288887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zh4ja1o/_remote_module_non_scriptable.py 2022-08-17T12:43:35.2899026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:35.2909869Z 2022-08-17T12:43:35.2910256Z Running tests... 2022-08-17T12:43:35.2911360Z ---------------------------------------------------------------------- 2022-08-17T12:43:35.5514204Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21100 2022-08-17T12:43:35.5541416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21101 2022-08-17T12:43:35.5569225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21102 2022-08-17T12:43:35.5601258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21103 2022-08-17T12:43:36.5719341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:36.5720020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:36.5722541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:36.5723231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:36.5895134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:36.5895748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:36.5898874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:36.5899536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:36.5999456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:36.6000137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:36.6003623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:36.6004272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:36.6015689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:36.6016254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:36.6020147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:36.6020739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:36.7294463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26yzli59 2022-08-17T12:43:36.7295292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26yzli59/_remote_module_non_scriptable.py 2022-08-17T12:43:36.7492858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwb3358r 2022-08-17T12:43:36.7493708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwb3358r/_remote_module_non_scriptable.py 2022-08-17T12:43:36.7598992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zfhmq7z 2022-08-17T12:43:36.7599676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zfhmq7z/_remote_module_non_scriptable.py 2022-08-17T12:43:36.7611995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppoijme1b 2022-08-17T12:43:36.7615070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppoijme1b/_remote_module_non_scriptable.py 2022-08-17T12:43:36.9910006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:37.0228315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:37.0241657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:37.0326395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:37.6658695Z ok (2.375s) 2022-08-17T12:43:37.6659330Z 2022-08-17T12:43:37.6659712Z ---------------------------------------------------------------------- 2022-08-17T12:43:37.6659978Z Ran 1 test in 2.375s 2022-08-17T12:43:37.6660100Z 2022-08-17T12:43:37.6660151Z OK 2022-08-17T12:43:37.6660248Z 2022-08-17T12:43:37.6660344Z Generating XML reports... 2022-08-17T12:43:37.6695955Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124335.xml 2022-08-17T12:43:38.9514682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:38.9515082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:38.9523551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:38.9524014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:39.1030474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphah7phkj 2022-08-17T12:43:39.1033696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphah7phkj/_remote_module_non_scriptable.py 2022-08-17T12:43:39.3641316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:39.3652373Z 2022-08-17T12:43:39.3652481Z Running tests... 2022-08-17T12:43:39.3652896Z ---------------------------------------------------------------------- 2022-08-17T12:43:39.6317258Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21323 2022-08-17T12:43:39.6345866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21324 2022-08-17T12:43:39.6373453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21325 2022-08-17T12:43:39.6404271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21326 2022-08-17T12:43:40.6423009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:40.6423666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:40.6426564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:40.6709979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:40.6710958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:40.6711592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:40.6712399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:40.6713011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:40.6715133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:40.6715817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:40.6716607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:40.6717227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:40.6824320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:40.6828301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:40.6829198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:40.6829834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:40.7991294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7q8uqa_ 2022-08-17T12:43:40.7992360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7q8uqa_/_remote_module_non_scriptable.py 2022-08-17T12:43:40.8304997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwjb8et0 2022-08-17T12:43:40.8305788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwjb8et0/_remote_module_non_scriptable.py 2022-08-17T12:43:40.8309957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyends1g 2022-08-17T12:43:40.8312009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyends1g/_remote_module_non_scriptable.py 2022-08-17T12:43:40.8417842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ow5uodn 2022-08-17T12:43:40.8419097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ow5uodn/_remote_module_non_scriptable.py 2022-08-17T12:43:41.0578096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:41.0885802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:41.0949272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:41.1040620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:41.7463603Z ok (2.381s) 2022-08-17T12:43:41.7463903Z 2022-08-17T12:43:41.7464436Z ---------------------------------------------------------------------- 2022-08-17T12:43:41.7464737Z Ran 1 test in 2.381s 2022-08-17T12:43:41.7464862Z 2022-08-17T12:43:41.7464926Z OK 2022-08-17T12:43:41.7465008Z 2022-08-17T12:43:41.7465108Z Generating XML reports... 2022-08-17T12:43:41.7497744Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124339.xml 2022-08-17T12:43:43.0500214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:43.0500921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:43.0503529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:43.0504131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:43.2012460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p7y1oim 2022-08-17T12:43:43.2016105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p7y1oim/_remote_module_non_scriptable.py 2022-08-17T12:43:43.4645720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:43.4656490Z 2022-08-17T12:43:43.4656567Z Running tests... 2022-08-17T12:43:43.4657348Z ---------------------------------------------------------------------- 2022-08-17T12:43:43.7269192Z test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21546 2022-08-17T12:43:43.7297420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21547 2022-08-17T12:43:43.7326143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21548 2022-08-17T12:43:43.7357938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21549 2022-08-17T12:43:44.8310573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:44.8311174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:44.8314374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:44.8315061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:44.8434734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:44.8435672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:44.8439006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:44.8439652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:44.8789225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:44.8789867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:44.8792771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:44.8793453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:44.8915804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:44.8916508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:44.8920029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:44.8920643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:44.9889572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkge81u95 2022-08-17T12:43:44.9890745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkge81u95/_remote_module_non_scriptable.py 2022-08-17T12:43:45.0016661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0mw489t 2022-08-17T12:43:45.0017711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0mw489t/_remote_module_non_scriptable.py 2022-08-17T12:43:45.0380927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xf4kzty 2022-08-17T12:43:45.0381889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xf4kzty/_remote_module_non_scriptable.py 2022-08-17T12:43:45.0471675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zg0ywv0 2022-08-17T12:43:45.0474025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zg0ywv0/_remote_module_non_scriptable.py 2022-08-17T12:43:45.2476084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:45.2627601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:45.2964215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:45.3046985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:45.9417559Z ok (2.476s) 2022-08-17T12:43:45.9417716Z 2022-08-17T12:43:45.9418041Z ---------------------------------------------------------------------- 2022-08-17T12:43:45.9418364Z Ran 1 test in 2.476s 2022-08-17T12:43:45.9418483Z 2022-08-17T12:43:45.9418561Z OK 2022-08-17T12:43:45.9418655Z 2022-08-17T12:43:45.9418747Z Generating XML reports... 2022-08-17T12:43:45.9451424Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124343.xml 2022-08-17T12:43:47.2405015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:47.2405472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:47.2408365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:47.2408744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:47.3929288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmemt0_n 2022-08-17T12:43:47.3932464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmemt0_n/_remote_module_non_scriptable.py 2022-08-17T12:43:47.6555740Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:47.6566962Z 2022-08-17T12:43:47.6567106Z Running tests... 2022-08-17T12:43:47.6568166Z ---------------------------------------------------------------------- 2022-08-17T12:43:47.9199403Z test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21785 2022-08-17T12:43:47.9226741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21786 2022-08-17T12:43:47.9255419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21787 2022-08-17T12:43:47.9286112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21788 2022-08-17T12:43:48.9830235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:48.9830864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:48.9834078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:48.9834802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:49.0180286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:49.0180960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:49.0183916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:49.0184566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:49.0378231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:49.0378893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:49.0382277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:49.0382938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:49.0717516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:49.0718121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:49.0721195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:49.0721789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:49.1447626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnpphn28 2022-08-17T12:43:49.1448331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnpphn28/_remote_module_non_scriptable.py 2022-08-17T12:43:49.1764996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparxzc9w4 2022-08-17T12:43:49.1766226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparxzc9w4/_remote_module_non_scriptable.py 2022-08-17T12:43:49.1968585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dbi2myk 2022-08-17T12:43:49.1969447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dbi2myk/_remote_module_non_scriptable.py 2022-08-17T12:43:49.2306295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76umhoo2 2022-08-17T12:43:49.2307140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76umhoo2/_remote_module_non_scriptable.py 2022-08-17T12:43:49.4083546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:49.4367218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:49.4580844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:49.4943552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:50.2348540Z ok (2.578s) 2022-08-17T12:43:50.2348912Z 2022-08-17T12:43:50.2349374Z ---------------------------------------------------------------------- 2022-08-17T12:43:50.2349700Z Ran 1 test in 2.578s 2022-08-17T12:43:50.2349839Z 2022-08-17T12:43:50.2349908Z OK 2022-08-17T12:43:50.2350001Z 2022-08-17T12:43:50.2350093Z Generating XML reports... 2022-08-17T12:43:50.2383183Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124347.xml 2022-08-17T12:43:51.5437661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:51.5438144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:51.5439531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:51.5440052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:51.6950873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_vz7hu6 2022-08-17T12:43:51.6954590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_vz7hu6/_remote_module_non_scriptable.py 2022-08-17T12:43:51.9587178Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:51.9597773Z 2022-08-17T12:43:51.9598078Z Running tests... 2022-08-17T12:43:51.9598720Z ---------------------------------------------------------------------- 2022-08-17T12:43:52.2222600Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22024 2022-08-17T12:43:52.2252978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22025 2022-08-17T12:43:52.2282538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22026 2022-08-17T12:43:52.2313128Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22027 2022-08-17T12:43:53.2953919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:53.2954604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:53.2957695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:53.2958380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:53.3243091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:53.3243710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:53.3247114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:53.3247742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:53.3337202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:53.3337880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:53.3342157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:53.3342789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:53.3646978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:53.3647640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:53.3650938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:53.3651619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:53.4522895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyoioins 2022-08-17T12:43:53.4523981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyoioins/_remote_module_non_scriptable.py 2022-08-17T12:43:53.4816544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5hfs0zy 2022-08-17T12:43:53.4817641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5hfs0zy/_remote_module_non_scriptable.py 2022-08-17T12:43:53.4917836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjtyc22x 2022-08-17T12:43:53.4919479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjtyc22x/_remote_module_non_scriptable.py 2022-08-17T12:43:53.5203465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjy6bt6w 2022-08-17T12:43:53.5204347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjy6bt6w/_remote_module_non_scriptable.py 2022-08-17T12:43:53.7206709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:53.7405235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:53.7752343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:53.7855135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:54.1368984Z ok (2.177s) 2022-08-17T12:43:54.1369237Z 2022-08-17T12:43:54.1369775Z ---------------------------------------------------------------------- 2022-08-17T12:43:54.1370151Z Ran 1 test in 2.177s 2022-08-17T12:43:54.1370278Z 2022-08-17T12:43:54.1370327Z OK 2022-08-17T12:43:54.1370426Z 2022-08-17T12:43:54.1370525Z Generating XML reports... 2022-08-17T12:43:54.1404017Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124351.xml 2022-08-17T12:43:55.4339597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:55.4340082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:55.4342887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:55.4343297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:55.5860277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjkx42kt 2022-08-17T12:43:55.5863462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjkx42kt/_remote_module_non_scriptable.py 2022-08-17T12:43:55.8569021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:43:55.8580020Z 2022-08-17T12:43:55.8580132Z Running tests... 2022-08-17T12:43:55.8581126Z ---------------------------------------------------------------------- 2022-08-17T12:43:56.1261455Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22095 2022-08-17T12:43:56.1289082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22096 2022-08-17T12:43:56.1319186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22097 2022-08-17T12:43:56.1351962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22098 2022-08-17T12:43:57.2067948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:57.2068598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:57.2071088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:57.2071774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:57.2290448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:57.2291131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:57.2295082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:57.2295768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:57.2398381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:57.2399010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:57.2402867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:57.2403576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:57.2583646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:57.2584196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:57.2587376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:57.2587921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:57.3659747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ioiywun 2022-08-17T12:43:57.3660643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ioiywun/_remote_module_non_scriptable.py 2022-08-17T12:43:57.3876378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8fgbnhf 2022-08-17T12:43:57.3879469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8fgbnhf/_remote_module_non_scriptable.py 2022-08-17T12:43:57.4008355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjaheigka 2022-08-17T12:43:57.4009974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjaheigka/_remote_module_non_scriptable.py 2022-08-17T12:43:57.4140808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sgm03xo 2022-08-17T12:43:57.4141992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sgm03xo/_remote_module_non_scriptable.py 2022-08-17T12:43:57.6239591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:43:57.6448460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:43:57.6621331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:43:57.6728509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:43:58.3412875Z ok (2.483s) 2022-08-17T12:43:58.3413169Z 2022-08-17T12:43:58.3413744Z ---------------------------------------------------------------------- 2022-08-17T12:43:58.3414189Z Ran 1 test in 2.483s 2022-08-17T12:43:58.3414311Z 2022-08-17T12:43:58.3414378Z OK 2022-08-17T12:43:58.3414460Z 2022-08-17T12:43:58.3414556Z Generating XML reports... 2022-08-17T12:43:58.3448296Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124355.xml 2022-08-17T12:43:59.6383932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:43:59.6384331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:43:59.6387029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:43:59.6387417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:43:59.7920982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85tnh518 2022-08-17T12:43:59.7924229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85tnh518/_remote_module_non_scriptable.py 2022-08-17T12:44:00.0583437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:00.0594534Z 2022-08-17T12:44:00.0594651Z Running tests... 2022-08-17T12:44:00.0595293Z ---------------------------------------------------------------------- 2022-08-17T12:44:00.3246718Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22322 2022-08-17T12:44:00.3277488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22323 2022-08-17T12:44:00.3306806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22324 2022-08-17T12:44:00.3337061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22325 2022-08-17T12:44:01.4358062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:01.4358769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:01.4361513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:01.4362445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:01.4373131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:01.4373741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:01.4374482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:01.4375085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:01.4378595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:01.4379279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:01.4380095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:01.4380714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:01.4565499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:01.4566355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:01.4568840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:01.4569441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:01.5942562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuho5ee0y 2022-08-17T12:44:01.5944103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuho5ee0y/_remote_module_non_scriptable.py 2022-08-17T12:44:01.5976793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvt21v2x 2022-08-17T12:44:01.5978322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4y8q4z8 2022-08-17T12:44:01.5980029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvt21v2x/_remote_module_non_scriptable.py 2022-08-17T12:44:01.5981132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4y8q4z8/_remote_module_non_scriptable.py 2022-08-17T12:44:01.6147786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5wgd051 2022-08-17T12:44:01.6149053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5wgd051/_remote_module_non_scriptable.py 2022-08-17T12:44:01.8569537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:01.8624293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:01.8636848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:01.8794986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:02.5398040Z ok (2.480s) 2022-08-17T12:44:02.5398308Z 2022-08-17T12:44:02.5398840Z ---------------------------------------------------------------------- 2022-08-17T12:44:02.5399261Z Ran 1 test in 2.480s 2022-08-17T12:44:02.5399371Z 2022-08-17T12:44:02.5399440Z OK 2022-08-17T12:44:02.5399534Z 2022-08-17T12:44:02.5399637Z Generating XML reports... 2022-08-17T12:44:02.5434776Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124400.xml 2022-08-17T12:44:03.8325514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:03.8326107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:03.8328361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:03.8328736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:03.9815211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ygou15m 2022-08-17T12:44:03.9817567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ygou15m/_remote_module_non_scriptable.py 2022-08-17T12:44:04.2463127Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:04.2474206Z 2022-08-17T12:44:04.2474328Z Running tests... 2022-08-17T12:44:04.2474917Z ---------------------------------------------------------------------- 2022-08-17T12:44:04.5091603Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22549 2022-08-17T12:44:04.5120713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22550 2022-08-17T12:44:04.5150401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22551 2022-08-17T12:44:04.5181330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22552 2022-08-17T12:44:05.4903317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:05.4903970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:05.4905751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:05.4906497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:05.4914895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:05.4915697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:05.4919062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:05.4919705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:05.5070499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:05.5071129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:05.5073824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:05.5074466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:05.5496574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:05.5497184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:05.5500020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:05.5500598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:05.6493361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnkxaj8a 2022-08-17T12:44:05.6494362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnkxaj8a/_remote_module_non_scriptable.py 2022-08-17T12:44:05.6528079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5zwpd1k 2022-08-17T12:44:05.6529553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5zwpd1k/_remote_module_non_scriptable.py 2022-08-17T12:44:05.6658987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcqrop0i 2022-08-17T12:44:05.6660147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcqrop0i/_remote_module_non_scriptable.py 2022-08-17T12:44:05.7122889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph87t294u 2022-08-17T12:44:05.7129088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph87t294u/_remote_module_non_scriptable.py 2022-08-17T12:44:05.9105662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:05.9170302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:05.9277819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:05.9710954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:06.7242172Z ok (2.476s) 2022-08-17T12:44:06.7242397Z 2022-08-17T12:44:06.7242804Z ---------------------------------------------------------------------- 2022-08-17T12:44:06.7243057Z Ran 1 test in 2.477s 2022-08-17T12:44:06.7243233Z 2022-08-17T12:44:06.7243322Z OK 2022-08-17T12:44:06.7243419Z 2022-08-17T12:44:06.7243516Z Generating XML reports... 2022-08-17T12:44:06.7276561Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124404.xml 2022-08-17T12:44:08.0129782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:08.0130188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:08.0132151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:08.0132549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:08.1621666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwx6t61pz 2022-08-17T12:44:08.1624163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwx6t61pz/_remote_module_non_scriptable.py 2022-08-17T12:44:08.4220730Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:08.4232005Z 2022-08-17T12:44:08.4232381Z Running tests... 2022-08-17T12:44:08.4232806Z ---------------------------------------------------------------------- 2022-08-17T12:44:08.6843182Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22776 2022-08-17T12:44:08.6871262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22777 2022-08-17T12:44:08.6899873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22778 2022-08-17T12:44:08.6931185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22779 2022-08-17T12:44:09.7204449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:09.7205039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:09.7207633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:09.7208229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:09.7295036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:09.7295686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:09.7298995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:09.7299693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:09.7301051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:09.7301619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:09.7306521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:09.7307207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:09.7445415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:09.7446319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:09.7448815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:09.7449462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:09.8812976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zchoo2p 2022-08-17T12:44:09.8813779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zchoo2p/_remote_module_non_scriptable.py 2022-08-17T12:44:09.8888670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx98gi6ea 2022-08-17T12:44:09.8889639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx98gi6ea/_remote_module_non_scriptable.py 2022-08-17T12:44:09.8898505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3blm255 2022-08-17T12:44:09.8900033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3blm255/_remote_module_non_scriptable.py 2022-08-17T12:44:09.9030051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpwdbq31 2022-08-17T12:44:09.9031252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpwdbq31/_remote_module_non_scriptable.py 2022-08-17T12:44:10.1435067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:10.1479135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:10.1488654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:10.1626986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:10.9994879Z ok (2.576s) 2022-08-17T12:44:10.9995115Z 2022-08-17T12:44:10.9995615Z ---------------------------------------------------------------------- 2022-08-17T12:44:10.9996080Z Ran 1 test in 2.576s 2022-08-17T12:44:10.9996203Z 2022-08-17T12:44:10.9996267Z OK 2022-08-17T12:44:10.9996363Z 2022-08-17T12:44:10.9996461Z Generating XML reports... 2022-08-17T12:44:11.0030170Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124408.xml 2022-08-17T12:44:12.3062902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:12.3063375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:12.3066008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:12.3066744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:12.4743901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ob2o0rk 2022-08-17T12:44:12.4747088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ob2o0rk/_remote_module_non_scriptable.py 2022-08-17T12:44:12.7387280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:12.7397776Z 2022-08-17T12:44:12.7397877Z Running tests... 2022-08-17T12:44:12.7398808Z ---------------------------------------------------------------------- 2022-08-17T12:44:13.0030878Z test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23003 2022-08-17T12:44:13.0058970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23004 2022-08-17T12:44:13.0088161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23005 2022-08-17T12:44:13.0118574Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23006 2022-08-17T12:44:14.0424260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:14.0425190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:14.0426602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:14.0427238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:14.1462060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:14.1462746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:14.1463581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:14.1464153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:14.1465313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:14.1466009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:14.1466825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:14.1467397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:14.2007125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpui5h2kmn 2022-08-17T12:44:14.2007993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpui5h2kmn/_remote_module_non_scriptable.py 2022-08-17T12:44:14.2245693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:14.2246600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:14.2248740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:14.2249405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:14.3038753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7z49rc2 2022-08-17T12:44:14.3039583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7z49rc2/_remote_module_non_scriptable.py 2022-08-17T12:44:14.3047292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmzs8lyv 2022-08-17T12:44:14.3049191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmzs8lyv/_remote_module_non_scriptable.py 2022-08-17T12:44:14.3813077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nvpe5q5 2022-08-17T12:44:14.3813841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nvpe5q5/_remote_module_non_scriptable.py 2022-08-17T12:44:14.4664417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:14.5640749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:14.5871743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:14.6423054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:15.3179005Z ok (2.578s) 2022-08-17T12:44:15.3179271Z 2022-08-17T12:44:15.3179837Z ---------------------------------------------------------------------- 2022-08-17T12:44:15.3180292Z Ran 1 test in 2.578s 2022-08-17T12:44:15.3180445Z 2022-08-17T12:44:15.3180511Z OK 2022-08-17T12:44:15.3180608Z 2022-08-17T12:44:15.3180728Z Generating XML reports... 2022-08-17T12:44:15.3214455Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124412.xml 2022-08-17T12:44:16.6154163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:16.6154564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:16.6157253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:16.6157640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:16.7680695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pynl52b 2022-08-17T12:44:16.7683456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pynl52b/_remote_module_non_scriptable.py 2022-08-17T12:44:17.0327665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:17.0338280Z 2022-08-17T12:44:17.0338381Z Running tests... 2022-08-17T12:44:17.0338801Z ---------------------------------------------------------------------- 2022-08-17T12:44:17.2967896Z test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23226 2022-08-17T12:44:17.2996590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23227 2022-08-17T12:44:17.3027871Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23228 2022-08-17T12:44:17.3062104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23229 2022-08-17T12:44:18.2892774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:18.2893395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:18.2896632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:18.2897303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:18.3269943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:18.3270602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:18.3273666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:18.3274349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:18.3558135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:18.3558786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:18.3562493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:18.3563148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:18.3607121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:18.3608038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:18.3612506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:18.3613209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:18.4486693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp512ctg2b 2022-08-17T12:44:18.4487909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp512ctg2b/_remote_module_non_scriptable.py 2022-08-17T12:44:18.4876207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv0nlrx3 2022-08-17T12:44:18.4877051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv0nlrx3/_remote_module_non_scriptable.py 2022-08-17T12:44:18.5169053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjym06jff 2022-08-17T12:44:18.5169873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjym06jff/_remote_module_non_scriptable.py 2022-08-17T12:44:18.5183562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupoiy9u3 2022-08-17T12:44:18.5185695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupoiy9u3/_remote_module_non_scriptable.py 2022-08-17T12:44:18.7108956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:18.7519572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:18.7753792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:18.8008626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:19.5123604Z ok (2.478s) 2022-08-17T12:44:19.5123832Z 2022-08-17T12:44:19.5124293Z ---------------------------------------------------------------------- 2022-08-17T12:44:19.5124729Z Ran 1 test in 2.478s 2022-08-17T12:44:19.5124895Z 2022-08-17T12:44:19.5124961Z OK 2022-08-17T12:44:19.5125086Z 2022-08-17T12:44:19.5125213Z Generating XML reports... 2022-08-17T12:44:19.5160142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124417.xml 2022-08-17T12:44:20.8176728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:20.8177121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:20.8178717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:20.8179091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:20.9708472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3vlhrs5 2022-08-17T12:44:20.9711652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3vlhrs5/_remote_module_non_scriptable.py 2022-08-17T12:44:21.2358168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:21.2369381Z 2022-08-17T12:44:21.2369584Z Running tests... 2022-08-17T12:44:21.2370005Z ---------------------------------------------------------------------- 2022-08-17T12:44:21.5031411Z test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23465 2022-08-17T12:44:21.5060382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23466 2022-08-17T12:44:21.5088869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23467 2022-08-17T12:44:21.5120146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23468 2022-08-17T12:44:22.5969770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:22.5970768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:22.5973203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:22.5973875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:22.6045758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:22.6046602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:22.6050303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:22.6050923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:22.6419762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:22.6420414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:22.6423422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:22.6424015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:22.6505773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:22.6506524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:22.6509386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:22.6509922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:22.7557709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbqs8mcn 2022-08-17T12:44:22.7558496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbqs8mcn/_remote_module_non_scriptable.py 2022-08-17T12:44:22.7646233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoke0sfke 2022-08-17T12:44:22.7647200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoke0sfke/_remote_module_non_scriptable.py 2022-08-17T12:44:22.8058335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqornpcp 2022-08-17T12:44:22.8059132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqornpcp/_remote_module_non_scriptable.py 2022-08-17T12:44:22.8262904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns_3gr0a 2022-08-17T12:44:22.8263646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns_3gr0a/_remote_module_non_scriptable.py 2022-08-17T12:44:23.0203579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:23.0260330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:23.0619222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:23.0867781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:23.7180883Z ok (2.481s) 2022-08-17T12:44:23.7181147Z 2022-08-17T12:44:23.7181680Z ---------------------------------------------------------------------- 2022-08-17T12:44:23.7182126Z Ran 1 test in 2.481s 2022-08-17T12:44:23.7182253Z 2022-08-17T12:44:23.7182318Z OK 2022-08-17T12:44:23.7182419Z 2022-08-17T12:44:23.7182518Z Generating XML reports... 2022-08-17T12:44:23.7215456Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124421.xml 2022-08-17T12:44:25.0151469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:25.0151895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:25.0154845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:25.0155220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:25.1687609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn64k2bco 2022-08-17T12:44:25.1690697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn64k2bco/_remote_module_non_scriptable.py 2022-08-17T12:44:25.4352472Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:25.4362752Z 2022-08-17T12:44:25.4362831Z Running tests... 2022-08-17T12:44:25.4363850Z ---------------------------------------------------------------------- 2022-08-17T12:44:25.7015538Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23704 2022-08-17T12:44:25.7043389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23705 2022-08-17T12:44:25.7072217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23706 2022-08-17T12:44:25.7102814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23707 2022-08-17T12:44:26.7203002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:26.7203606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:26.7207191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:26.7207864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:26.7578608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:26.7579248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:26.7581723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:26.7582344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:26.7597835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:26.7598416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:26.7601997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:26.7602605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:26.7642733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:26.7643448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:26.7647655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:26.7648313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:26.8772868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jhc_gho 2022-08-17T12:44:26.8773663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jhc_gho/_remote_module_non_scriptable.py 2022-08-17T12:44:26.9192030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgpasmgg 2022-08-17T12:44:26.9193630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgpasmgg/_remote_module_non_scriptable.py 2022-08-17T12:44:26.9206921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6ccmmbh 2022-08-17T12:44:26.9208861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6ccmmbh/_remote_module_non_scriptable.py 2022-08-17T12:44:26.9418438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjp_xv4l 2022-08-17T12:44:26.9419508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjp_xv4l/_remote_module_non_scriptable.py 2022-08-17T12:44:27.1402818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:27.1847537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:27.1858659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:27.2064447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:27.8161452Z ok (2.380s) 2022-08-17T12:44:27.8167497Z 2022-08-17T12:44:27.8168167Z ---------------------------------------------------------------------- 2022-08-17T12:44:27.8168661Z Ran 1 test in 2.380s 2022-08-17T12:44:27.8168796Z 2022-08-17T12:44:27.8168850Z OK 2022-08-17T12:44:27.8168952Z 2022-08-17T12:44:27.8169063Z Generating XML reports... 2022-08-17T12:44:27.8199816Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124425.xml 2022-08-17T12:44:29.1104282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:29.1104947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:29.1107053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:29.1107439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:29.2611224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_tvdw3r 2022-08-17T12:44:29.2614676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_tvdw3r/_remote_module_non_scriptable.py 2022-08-17T12:44:29.5223642Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:29.5234450Z 2022-08-17T12:44:29.5234571Z Running tests... 2022-08-17T12:44:29.5235008Z ---------------------------------------------------------------------- 2022-08-17T12:44:29.7885505Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23927 2022-08-17T12:44:29.7914587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23928 2022-08-17T12:44:29.7943565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23929 2022-08-17T12:44:29.7976108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23930 2022-08-17T12:44:30.8459189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:30.8459863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:30.8462665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:30.8463291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:30.8587506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:30.8588197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:30.8591444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:30.8592098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:30.8917152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:30.8917705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:30.8921243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:30.8922194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:30.8953211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:30.8953986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:30.8957030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:30.8957836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:31.0060770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj3hbuem 2022-08-17T12:44:31.0061596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj3hbuem/_remote_module_non_scriptable.py 2022-08-17T12:44:31.0192360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8lpc23f 2022-08-17T12:44:31.0193728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8lpc23f/_remote_module_non_scriptable.py 2022-08-17T12:44:31.0543146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfgr7_4m 2022-08-17T12:44:31.0544236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl67857g6 2022-08-17T12:44:31.0545465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfgr7_4m/_remote_module_non_scriptable.py 2022-08-17T12:44:31.0546509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl67857g6/_remote_module_non_scriptable.py 2022-08-17T12:44:31.2732717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:31.3085098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:31.3214437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:31.3230126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:31.9035194Z ok (2.380s) 2022-08-17T12:44:31.9035438Z 2022-08-17T12:44:31.9035771Z ---------------------------------------------------------------------- 2022-08-17T12:44:31.9036089Z Ran 1 test in 2.380s 2022-08-17T12:44:31.9036216Z 2022-08-17T12:44:31.9036281Z OK 2022-08-17T12:44:31.9036394Z 2022-08-17T12:44:31.9036491Z Generating XML reports... 2022-08-17T12:44:31.9070281Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124429.xml 2022-08-17T12:44:33.2050851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:33.2051538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:33.2054363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:33.2054929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:33.3546311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdze6472f 2022-08-17T12:44:33.3550279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdze6472f/_remote_module_non_scriptable.py 2022-08-17T12:44:33.6181907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:33.6193210Z 2022-08-17T12:44:33.6193526Z Running tests... 2022-08-17T12:44:33.8866303Z ---------------------------------------------------------------------- 2022-08-17T12:44:33.8867122Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24150 2022-08-17T12:44:33.8895597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24151 2022-08-17T12:44:33.8924931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24152 2022-08-17T12:44:33.8957407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24153 2022-08-17T12:44:34.9334423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:34.9335111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:34.9337310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:34.9337983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:34.9342243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:34.9343009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:34.9346519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:34.9347177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:34.9506937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:34.9507586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:34.9511071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:34.9511756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:34.9692254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:34.9692904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:34.9695204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:34.9695868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:35.0948980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadh2zz79 2022-08-17T12:44:35.0949871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufsvhyos 2022-08-17T12:44:35.0950594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadh2zz79/_remote_module_non_scriptable.py 2022-08-17T12:44:35.0953492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufsvhyos/_remote_module_non_scriptable.py 2022-08-17T12:44:35.1108668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_xu7dn3 2022-08-17T12:44:35.1109901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_xu7dn3/_remote_module_non_scriptable.py 2022-08-17T12:44:35.1277440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qgugizo 2022-08-17T12:44:35.1278250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qgugizo/_remote_module_non_scriptable.py 2022-08-17T12:44:35.3576118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:35.3576864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:35.3715260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:35.4100830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:36.1017465Z ok (2.482s) 2022-08-17T12:44:36.1017635Z 2022-08-17T12:44:36.1018054Z ---------------------------------------------------------------------- 2022-08-17T12:44:36.1018337Z Ran 1 test in 2.482s 2022-08-17T12:44:36.1018461Z 2022-08-17T12:44:36.1018527Z OK 2022-08-17T12:44:36.1018639Z 2022-08-17T12:44:36.1018760Z Generating XML reports... 2022-08-17T12:44:36.1050990Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124433.xml 2022-08-17T12:44:37.4009354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:37.4009997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:37.4011959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:37.4012348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:37.5521431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q41mp1h 2022-08-17T12:44:37.5524706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q41mp1h/_remote_module_non_scriptable.py 2022-08-17T12:44:37.8158236Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:37.8169060Z 2022-08-17T12:44:37.8169161Z Running tests... 2022-08-17T12:44:37.8169650Z ---------------------------------------------------------------------- 2022-08-17T12:44:38.0819923Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24373 2022-08-17T12:44:38.0849018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24374 2022-08-17T12:44:38.0878762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24375 2022-08-17T12:44:38.0912228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24376 2022-08-17T12:44:39.0912240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:39.0912929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:39.0915807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:39.0916443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:39.0940487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:39.0941134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:39.0945347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:39.0946110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:39.1611397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:39.1612051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:39.1615441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:39.1616093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:39.1662060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:39.1662710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:39.1666716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:39.1667367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:39.2503792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqy6fpoo5 2022-08-17T12:44:39.2504583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy6fpoo5/_remote_module_non_scriptable.py 2022-08-17T12:44:39.2529424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bbx5nej 2022-08-17T12:44:39.2531956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bbx5nej/_remote_module_non_scriptable.py 2022-08-17T12:44:39.3209537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqeqjvfng 2022-08-17T12:44:39.3211172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqeqjvfng/_remote_module_non_scriptable.py 2022-08-17T12:44:39.3259076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_011k3zc 2022-08-17T12:44:39.3260649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_011k3zc/_remote_module_non_scriptable.py 2022-08-17T12:44:39.5141023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:39.5382000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:39.5856311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:39.5933478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:40.3977159Z ok (2.580s) 2022-08-17T12:44:40.3977429Z 2022-08-17T12:44:40.3977816Z ---------------------------------------------------------------------- 2022-08-17T12:44:40.3978102Z Ran 1 test in 2.581s 2022-08-17T12:44:40.3978224Z 2022-08-17T12:44:40.3978285Z OK 2022-08-17T12:44:40.3978381Z 2022-08-17T12:44:40.3978480Z Generating XML reports... 2022-08-17T12:44:40.4011434Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124437.xml 2022-08-17T12:44:41.6990568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:41.6990961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:41.6994224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:41.6994684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:41.8512587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvb78lgp 2022-08-17T12:44:41.8516289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvb78lgp/_remote_module_non_scriptable.py 2022-08-17T12:44:42.1168685Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:42.1180558Z 2022-08-17T12:44:42.1181054Z Running tests... 2022-08-17T12:44:42.1181752Z ---------------------------------------------------------------------- 2022-08-17T12:44:42.3818784Z test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24596 2022-08-17T12:44:42.3846813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24597 2022-08-17T12:44:42.3875873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24598 2022-08-17T12:44:42.3907716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24599 2022-08-17T12:44:43.4187729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:43.4188457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:43.4190635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:43.4191323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:43.4199877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:43.4200531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:43.4204167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:43.4204852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:43.4484442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:43.4485466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:43.4487999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:43.4489082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:43.4490101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:43.4490741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:43.4496927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:43.4497578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:43.5773294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhh_vz48 2022-08-17T12:44:43.5774484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhh_vz48/_remote_module_non_scriptable.py 2022-08-17T12:44:43.5784652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r4ir_c6 2022-08-17T12:44:43.5786931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r4ir_c6/_remote_module_non_scriptable.py 2022-08-17T12:44:43.6079261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxw76dzk 2022-08-17T12:44:43.6080699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxw76dzk/_remote_module_non_scriptable.py 2022-08-17T12:44:43.6293011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh60usu6 2022-08-17T12:44:43.6293893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh60usu6/_remote_module_non_scriptable.py 2022-08-17T12:44:43.8407528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:43.8422845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:43.8724484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:43.8928488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:44.5967587Z ok (2.478s) 2022-08-17T12:44:44.5967857Z 2022-08-17T12:44:44.5968353Z ---------------------------------------------------------------------- 2022-08-17T12:44:44.5968603Z Ran 1 test in 2.479s 2022-08-17T12:44:44.5968725Z 2022-08-17T12:44:44.5968789Z OK 2022-08-17T12:44:44.5968886Z 2022-08-17T12:44:44.5968982Z Generating XML reports... 2022-08-17T12:44:44.6001851Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124442.xml 2022-08-17T12:44:45.8874913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:45.8875319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:45.8878261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:45.8886375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:46.0402111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpaud50a 2022-08-17T12:44:46.0404838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpaud50a/_remote_module_non_scriptable.py 2022-08-17T12:44:46.3065179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:46.3075246Z 2022-08-17T12:44:46.3075384Z Running tests... 2022-08-17T12:44:46.3076118Z ---------------------------------------------------------------------- 2022-08-17T12:44:46.5721995Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24823 2022-08-17T12:44:46.5752122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24824 2022-08-17T12:44:46.5783011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24825 2022-08-17T12:44:46.5813906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24826 2022-08-17T12:44:47.6352817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:47.6353505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:47.6356273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:47.6356927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:47.6654891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:47.6655540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:47.6658028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:47.6658618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:47.6922622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:47.6923581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:47.6926336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:47.6926905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:47.7383924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:47.7384620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:47.7387780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:47.7388485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:47.7957228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7d8tdle 2022-08-17T12:44:47.7959130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7d8tdle/_remote_module_non_scriptable.py 2022-08-17T12:44:47.8312565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr3thyt_ 2022-08-17T12:44:47.8315121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr3thyt_/_remote_module_non_scriptable.py 2022-08-17T12:44:47.8676876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5qs7cpj 2022-08-17T12:44:47.8677897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5qs7cpj/_remote_module_non_scriptable.py 2022-08-17T12:44:47.9002878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmys139z 2022-08-17T12:44:47.9003647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmys139z/_remote_module_non_scriptable.py 2022-08-17T12:44:48.0570656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:48.0932608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:48.1283655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:48.1586477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:48.7874265Z ok (2.480s) 2022-08-17T12:44:48.7874513Z 2022-08-17T12:44:48.7874999Z ---------------------------------------------------------------------- 2022-08-17T12:44:48.7875422Z Ran 1 test in 2.480s 2022-08-17T12:44:48.7875618Z 2022-08-17T12:44:48.7875714Z OK 2022-08-17T12:44:48.7875845Z 2022-08-17T12:44:48.7875989Z Generating XML reports... 2022-08-17T12:44:48.7909464Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124446.xml 2022-08-17T12:44:50.0824612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:50.0825046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:50.0828211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:50.0828589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:50.2349001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pq1btr2 2022-08-17T12:44:50.2351913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pq1btr2/_remote_module_non_scriptable.py 2022-08-17T12:44:50.4971216Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:50.4982017Z 2022-08-17T12:44:50.4982396Z Running tests... 2022-08-17T12:44:50.4982824Z ---------------------------------------------------------------------- 2022-08-17T12:44:50.7595277Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25050 2022-08-17T12:44:50.7625129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25051 2022-08-17T12:44:50.7654562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25052 2022-08-17T12:44:50.7686829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25053 2022-08-17T12:44:51.8559870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:51.8560529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:51.8563341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:51.8564001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:51.8817912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:51.8818596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:51.8822216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:51.8822883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:51.8972655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:51.8973308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:51.8976089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:51.8976761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:51.9191445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:51.9192063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:51.9194892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:51.9195446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:52.0340056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_va3c0a 2022-08-17T12:44:52.0340924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_va3c0a/_remote_module_non_scriptable.py 2022-08-17T12:44:52.0410956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_uxn3ux 2022-08-17T12:44:52.0412384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_uxn3ux/_remote_module_non_scriptable.py 2022-08-17T12:44:52.0634550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_3a0_m_ 2022-08-17T12:44:52.0635527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_3a0_m_/_remote_module_non_scriptable.py 2022-08-17T12:44:52.0834310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7uyxo85 2022-08-17T12:44:52.0835984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7uyxo85/_remote_module_non_scriptable.py 2022-08-17T12:44:52.2935935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:52.3014925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:52.3276964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:52.3467393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:52.9747518Z ok (2.476s) 2022-08-17T12:44:52.9747804Z 2022-08-17T12:44:52.9748302Z ---------------------------------------------------------------------- 2022-08-17T12:44:52.9748636Z Ran 1 test in 2.476s 2022-08-17T12:44:52.9748887Z 2022-08-17T12:44:52.9749004Z OK 2022-08-17T12:44:52.9749184Z 2022-08-17T12:44:52.9749689Z Generating XML reports... 2022-08-17T12:44:52.9781471Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124450.xml 2022-08-17T12:44:54.2866965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:54.2867348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:54.2869725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:54.2870103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:54.4410075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesrsvtj2 2022-08-17T12:44:54.4413425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesrsvtj2/_remote_module_non_scriptable.py 2022-08-17T12:44:54.7075353Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:54.7086453Z 2022-08-17T12:44:54.7086601Z Running tests... 2022-08-17T12:44:54.7086972Z ---------------------------------------------------------------------- 2022-08-17T12:44:54.9776110Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25273 2022-08-17T12:44:54.9806134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25274 2022-08-17T12:44:54.9835377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25275 2022-08-17T12:44:54.9867616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25276 2022-08-17T12:44:56.0201146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:56.0201883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:56.0204465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:56.0205100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:56.0868955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:56.0869615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:56.0871997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:56.0872645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:56.0901806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:56.0902456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:56.0905919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:56.0906579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:56.1306284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:56.1306935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:56.1309851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:56.1310524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:56.1799594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslnrr6lf 2022-08-17T12:44:56.1800897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslnrr6lf/_remote_module_non_scriptable.py 2022-08-17T12:44:56.2481345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzewgo_t4 2022-08-17T12:44:56.2482298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzewgo_t4/_remote_module_non_scriptable.py 2022-08-17T12:44:56.2487437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dpwpb_e 2022-08-17T12:44:56.2490022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dpwpb_e/_remote_module_non_scriptable.py 2022-08-17T12:44:56.2880137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sh9c0yv 2022-08-17T12:44:56.2881008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sh9c0yv/_remote_module_non_scriptable.py 2022-08-17T12:44:56.4445428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:44:56.5095635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:44:56.5117556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:44:56.5486615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:44:57.2929007Z ok (2.584s) 2022-08-17T12:44:57.2929252Z 2022-08-17T12:44:57.2929805Z ---------------------------------------------------------------------- 2022-08-17T12:44:57.2930119Z Ran 1 test in 2.584s 2022-08-17T12:44:57.2930241Z 2022-08-17T12:44:57.2930314Z OK 2022-08-17T12:44:57.2930443Z 2022-08-17T12:44:57.2930573Z Generating XML reports... 2022-08-17T12:44:57.2964073Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124454.xml 2022-08-17T12:44:58.6121752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:44:58.6122473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:44:58.6125333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:44:58.6126159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:44:58.7632901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv12vso7 2022-08-17T12:44:58.7636176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv12vso7/_remote_module_non_scriptable.py 2022-08-17T12:44:59.0250692Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:44:59.0261742Z 2022-08-17T12:44:59.0262054Z Running tests... 2022-08-17T12:44:59.0262481Z ---------------------------------------------------------------------- 2022-08-17T12:44:59.2904178Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25496 2022-08-17T12:44:59.2931825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25497 2022-08-17T12:44:59.2960630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25498 2022-08-17T12:44:59.2992565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25499 2022-08-17T12:45:00.3484438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:00.3485054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:00.3487897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:00.3488528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:00.3566379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:00.3567121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:00.3571221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:00.3572138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:00.3613394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:00.3614060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:00.3618163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:00.3618789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:00.4177886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:00.4178498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:00.4181846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:00.4182525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:00.5094329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo9r02of 2022-08-17T12:45:00.5095147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo9r02of/_remote_module_non_scriptable.py 2022-08-17T12:45:00.5166546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthjf2vio 2022-08-17T12:45:00.5167850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthjf2vio/_remote_module_non_scriptable.py 2022-08-17T12:45:00.5209632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m0c_02f 2022-08-17T12:45:00.5211187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m0c_02f/_remote_module_non_scriptable.py 2022-08-17T12:45:00.5765511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmh0_kih 2022-08-17T12:45:00.5767019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmh0_kih/_remote_module_non_scriptable.py 2022-08-17T12:45:00.7715747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:00.7790714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:00.7811646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:00.8424682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:01.5053637Z ok (2.479s) 2022-08-17T12:45:01.5053867Z 2022-08-17T12:45:01.5054394Z ---------------------------------------------------------------------- 2022-08-17T12:45:01.5054860Z Ran 1 test in 2.479s 2022-08-17T12:45:01.5055424Z 2022-08-17T12:45:01.5055540Z OK 2022-08-17T12:45:01.5055689Z 2022-08-17T12:45:01.5055803Z Generating XML reports... 2022-08-17T12:45:01.5088574Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124459.xml 2022-08-17T12:45:02.8315203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:02.8315632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:02.8318333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:02.8318716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:02.9860868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk230_j9z 2022-08-17T12:45:02.9864234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk230_j9z/_remote_module_non_scriptable.py 2022-08-17T12:45:03.2535953Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:03.2547549Z 2022-08-17T12:45:03.2547664Z Running tests... 2022-08-17T12:45:03.2548168Z ---------------------------------------------------------------------- 2022-08-17T12:45:03.5219842Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25719 2022-08-17T12:45:03.5249255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25720 2022-08-17T12:45:03.5278814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25721 2022-08-17T12:45:03.5318697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25722 2022-08-17T12:45:04.6174889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:04.6175516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:04.6176697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:04.6177276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:04.6297195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:04.6297853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:04.6300619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:04.6301259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:04.6514353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:04.6514973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:04.6517760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:04.6518404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:04.6610277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:04.6610953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:04.6614984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:04.6615624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:04.7771533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnrqlh93 2022-08-17T12:45:04.7772372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnrqlh93/_remote_module_non_scriptable.py 2022-08-17T12:45:04.7891932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ayji61h 2022-08-17T12:45:04.7893253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ayji61h/_remote_module_non_scriptable.py 2022-08-17T12:45:04.8101765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjfrs4vq 2022-08-17T12:45:04.8103023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjfrs4vq/_remote_module_non_scriptable.py 2022-08-17T12:45:04.8218623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedj0ymn9 2022-08-17T12:45:04.8219620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedj0ymn9/_remote_module_non_scriptable.py 2022-08-17T12:45:05.0371501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:05.0495027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:05.0682463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:05.0844750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:05.7379327Z ok (2.483s) 2022-08-17T12:45:05.7379591Z 2022-08-17T12:45:05.7380061Z ---------------------------------------------------------------------- 2022-08-17T12:45:05.7380769Z Ran 1 test in 2.483s 2022-08-17T12:45:05.7380943Z 2022-08-17T12:45:05.7381047Z OK 2022-08-17T12:45:05.7381209Z 2022-08-17T12:45:05.7381356Z Generating XML reports... 2022-08-17T12:45:05.7414753Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124503.xml 2022-08-17T12:45:07.0545315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:07.0545778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:07.0548493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:07.0548969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:07.2074878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhg0a_to 2022-08-17T12:45:07.2078261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhg0a_to/_remote_module_non_scriptable.py 2022-08-17T12:45:07.4754798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:07.4765821Z 2022-08-17T12:45:07.4766133Z Running tests... 2022-08-17T12:45:07.4766770Z ---------------------------------------------------------------------- 2022-08-17T12:45:07.7460490Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25942 2022-08-17T12:45:07.7490855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25943 2022-08-17T12:45:07.7522944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25944 2022-08-17T12:45:07.7556020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25945 2022-08-17T12:45:08.8367646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:08.8368251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:08.8371268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:08.8371893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:08.8543632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:08.8544271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:08.8547374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:08.8548351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:08.8725804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:08.8726697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:08.8729668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:08.8730351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:08.8745350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:08.8746022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:08.8750243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:08.8751181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:08.9957923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtqljlal 2022-08-17T12:45:08.9958972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtqljlal/_remote_module_non_scriptable.py 2022-08-17T12:45:09.0147601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t22re47 2022-08-17T12:45:09.0148677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t22re47/_remote_module_non_scriptable.py 2022-08-17T12:45:09.0316105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7wkgstz 2022-08-17T12:45:09.0317217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7wkgstz/_remote_module_non_scriptable.py 2022-08-17T12:45:09.0330623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzk5hv0w 2022-08-17T12:45:09.0332267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzk5hv0w/_remote_module_non_scriptable.py 2022-08-17T12:45:09.2543713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:09.2749098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:09.2926913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:09.2927666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:09.9616388Z ok (2.485s) 2022-08-17T12:45:09.9616641Z 2022-08-17T12:45:09.9617168Z ---------------------------------------------------------------------- 2022-08-17T12:45:09.9617436Z Ran 1 test in 2.485s 2022-08-17T12:45:09.9617557Z 2022-08-17T12:45:09.9617621Z OK 2022-08-17T12:45:09.9617718Z 2022-08-17T12:45:09.9617801Z Generating XML reports... 2022-08-17T12:45:09.9651010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124507.xml 2022-08-17T12:45:11.2602371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:11.2602774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:11.2605413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:11.2605798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:11.4121975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6ja68ss 2022-08-17T12:45:11.4125193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6ja68ss/_remote_module_non_scriptable.py 2022-08-17T12:45:11.6760226Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:11.6771250Z 2022-08-17T12:45:11.6771392Z Running tests... 2022-08-17T12:45:11.6772287Z ---------------------------------------------------------------------- 2022-08-17T12:45:11.9481536Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26165 2022-08-17T12:45:11.9512029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26166 2022-08-17T12:45:11.9541461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26167 2022-08-17T12:45:11.9572597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26168 2022-08-17T12:45:13.0088704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:13.0089311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:13.0092170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:13.0092800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:13.0412396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:13.0413038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:13.0416780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:13.0417445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:13.0765758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:13.0766620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:13.0769346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:13.0769994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:13.0985655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:13.0986200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:13.0989354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:13.0990054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:13.1659363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeynyjyqw 2022-08-17T12:45:13.1660646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeynyjyqw/_remote_module_non_scriptable.py 2022-08-17T12:45:13.1997138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckco55ce 2022-08-17T12:45:13.1997980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckco55ce/_remote_module_non_scriptable.py 2022-08-17T12:45:13.2347366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_eqxup0q 2022-08-17T12:45:13.2348382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_eqxup0q/_remote_module_non_scriptable.py 2022-08-17T12:45:13.2578604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkvt_5v2 2022-08-17T12:45:13.2579919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkvt_5v2/_remote_module_non_scriptable.py 2022-08-17T12:45:13.4285118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:13.4618279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:13.4938370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:13.5187755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:14.2634010Z ok (2.586s) 2022-08-17T12:45:14.2634487Z 2022-08-17T12:45:14.2635116Z ---------------------------------------------------------------------- 2022-08-17T12:45:14.2635452Z Ran 1 test in 2.586s 2022-08-17T12:45:14.2635620Z 2022-08-17T12:45:14.2635693Z OK 2022-08-17T12:45:14.2635852Z 2022-08-17T12:45:14.2635948Z Generating XML reports... 2022-08-17T12:45:14.2669669Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124511.xml 2022-08-17T12:45:15.5829951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:15.5830336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:15.5832355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:15.5832814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:15.7348943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1i1kyts 2022-08-17T12:45:15.7351799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1i1kyts/_remote_module_non_scriptable.py 2022-08-17T12:45:16.0009819Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:16.0020159Z 2022-08-17T12:45:16.0020727Z Running tests... 2022-08-17T12:45:16.0021313Z ---------------------------------------------------------------------- 2022-08-17T12:45:16.2645282Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26392 2022-08-17T12:45:16.2674149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26393 2022-08-17T12:45:16.2703069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26394 2022-08-17T12:45:16.2735201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26395 2022-08-17T12:45:17.3025126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:17.3025840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:17.3028831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:17.3029433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:17.3117473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:17.3118115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:17.3121495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:17.3122173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:17.3234226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:17.3234858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:17.3238129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:17.3238837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:17.3313484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:17.3314150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:17.3318823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:17.3319501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:17.4624700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtfduzl4 2022-08-17T12:45:17.4625885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtfduzl4/_remote_module_non_scriptable.py 2022-08-17T12:45:17.4705542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz1bqsgh 2022-08-17T12:45:17.4707055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz1bqsgh/_remote_module_non_scriptable.py 2022-08-17T12:45:17.4821131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0_5yp2k 2022-08-17T12:45:17.4823288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0_5yp2k/_remote_module_non_scriptable.py 2022-08-17T12:45:17.4903237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfb_9ouy 2022-08-17T12:45:17.4904193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfb_9ouy/_remote_module_non_scriptable.py 2022-08-17T12:45:17.7222676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:17.7275671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:17.7389319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:17.7482679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:19.0802966Z ok (3.078s) 2022-08-17T12:45:19.0803267Z 2022-08-17T12:45:19.0803748Z ---------------------------------------------------------------------- 2022-08-17T12:45:19.0804000Z Ran 1 test in 3.078s 2022-08-17T12:45:19.0804121Z 2022-08-17T12:45:19.0804187Z OK 2022-08-17T12:45:19.0804285Z 2022-08-17T12:45:19.0805793Z Generating XML reports... 2022-08-17T12:45:19.0837496Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124515.xml 2022-08-17T12:45:20.3878274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:20.3878703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:20.3881115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:20.3881497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:20.5413196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwagfkhwi 2022-08-17T12:45:20.5416681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwagfkhwi/_remote_module_non_scriptable.py 2022-08-17T12:45:20.8065442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:20.8076521Z 2022-08-17T12:45:20.8076610Z Running tests... 2022-08-17T12:45:20.8077212Z ---------------------------------------------------------------------- 2022-08-17T12:45:21.0723246Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26447 2022-08-17T12:45:21.0753055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26448 2022-08-17T12:45:21.0782575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26449 2022-08-17T12:45:21.0813793Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26450 2022-08-17T12:45:22.1503581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:22.1504246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:22.1506426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:22.1507103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:22.1900736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:22.1901721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:22.1904104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:22.1904664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:22.1905568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:22.1906185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:22.1909096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:22.1909697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:22.2244899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:22.2245604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:22.2248779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:22.2249511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:22.3091679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc8ttyl1 2022-08-17T12:45:22.3093327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc8ttyl1/_remote_module_non_scriptable.py 2022-08-17T12:45:22.3486776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg1zze7k 2022-08-17T12:45:22.3488101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg1zze7k/_remote_module_non_scriptable.py 2022-08-17T12:45:22.3700504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1uw08t7 2022-08-17T12:45:22.3701564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1uw08t7/_remote_module_non_scriptable.py 2022-08-17T12:45:22.3882556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2qhkpcp 2022-08-17T12:45:22.3883691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2qhkpcp/_remote_module_non_scriptable.py 2022-08-17T12:45:22.5699712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:22.6086924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:22.6285413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:22.6462402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:23.2873427Z ok (2.479s) 2022-08-17T12:45:23.2873705Z 2022-08-17T12:45:23.2874081Z ---------------------------------------------------------------------- 2022-08-17T12:45:23.2874455Z Ran 1 test in 2.480s 2022-08-17T12:45:23.2874661Z 2022-08-17T12:45:23.2874779Z OK 2022-08-17T12:45:23.2874963Z 2022-08-17T12:45:23.2875138Z Generating XML reports... 2022-08-17T12:45:23.2907834Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124520.xml 2022-08-17T12:45:24.5947246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:24.5947651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:24.5950107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:24.5950554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:24.7475493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwnf42u2 2022-08-17T12:45:24.7478939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwnf42u2/_remote_module_non_scriptable.py 2022-08-17T12:45:25.0136198Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:25.0147539Z 2022-08-17T12:45:25.0147753Z Running tests... 2022-08-17T12:45:25.0148230Z ---------------------------------------------------------------------- 2022-08-17T12:45:25.2784001Z test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26674 2022-08-17T12:45:25.2811350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26675 2022-08-17T12:45:25.2840301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26676 2022-08-17T12:45:25.2872512Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26677 2022-08-17T12:45:26.3186652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:26.3187327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:26.3189461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:26.3190144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:26.3289459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:26.3290138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:26.3293467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:26.3294138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:26.3501815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:26.3502479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:26.3505392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:26.3506176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:26.3518726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:26.3519388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:26.3524268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:26.3524944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:26.4777791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq6aghs9 2022-08-17T12:45:26.4778689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq6aghs9/_remote_module_non_scriptable.py 2022-08-17T12:45:26.4874755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesw6h1ue 2022-08-17T12:45:26.4876155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesw6h1ue/_remote_module_non_scriptable.py 2022-08-17T12:45:26.5181949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsx2y66on 2022-08-17T12:45:26.5182704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsx2y66on/_remote_module_non_scriptable.py 2022-08-17T12:45:26.5270390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xswz9o3 2022-08-17T12:45:26.5272526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xswz9o3/_remote_module_non_scriptable.py 2022-08-17T12:45:26.7388407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:26.7459344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:26.7810696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:26.7897522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:27.5935949Z ok (2.578s) 2022-08-17T12:45:27.5936245Z 2022-08-17T12:45:27.5936774Z ---------------------------------------------------------------------- 2022-08-17T12:45:27.5937176Z Ran 1 test in 2.579s 2022-08-17T12:45:27.5937318Z 2022-08-17T12:45:27.5937383Z OK 2022-08-17T12:45:27.5938019Z 2022-08-17T12:45:27.5938154Z Generating XML reports... 2022-08-17T12:45:27.5971820Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124525.xml 2022-08-17T12:45:28.8919969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:28.8920346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:28.8922758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:28.8923158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:29.0416434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyliolk6 2022-08-17T12:45:29.0419902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyliolk6/_remote_module_non_scriptable.py 2022-08-17T12:45:29.3069868Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:29.3080443Z 2022-08-17T12:45:29.3080612Z Running tests... 2022-08-17T12:45:29.3080967Z ---------------------------------------------------------------------- 2022-08-17T12:45:29.5757617Z test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26897 2022-08-17T12:45:29.5785345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26898 2022-08-17T12:45:29.5814275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26899 2022-08-17T12:45:29.5846168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26900 2022-08-17T12:45:30.5833053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:30.5833764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:30.5836768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:30.5837393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:30.6107634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:30.6108253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:30.6110896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:30.6111583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:30.6370968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:30.6371619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:30.6374832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:30.6375433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:30.6441977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:30.6442655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:30.6446911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:30.6447544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:30.7423243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqnc7hdh 2022-08-17T12:45:30.7424053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqnc7hdh/_remote_module_non_scriptable.py 2022-08-17T12:45:30.7701834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2tvhtph 2022-08-17T12:45:30.7702729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2tvhtph/_remote_module_non_scriptable.py 2022-08-17T12:45:30.7964159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsma_b9_b 2022-08-17T12:45:30.7965551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsma_b9_b/_remote_module_non_scriptable.py 2022-08-17T12:45:30.8050075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge_pp55j 2022-08-17T12:45:30.8051313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge_pp55j/_remote_module_non_scriptable.py 2022-08-17T12:45:31.0042052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:31.0346744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:31.0599736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:31.0677911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:31.7906993Z ok (2.482s) 2022-08-17T12:45:31.7907173Z 2022-08-17T12:45:31.7907638Z ---------------------------------------------------------------------- 2022-08-17T12:45:31.7908055Z Ran 1 test in 2.483s 2022-08-17T12:45:31.7908273Z 2022-08-17T12:45:31.7908373Z OK 2022-08-17T12:45:31.7908554Z 2022-08-17T12:45:31.7908715Z Generating XML reports... 2022-08-17T12:45:31.7943618Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124529.xml 2022-08-17T12:45:33.1024321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:33.1024730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:33.1027495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:33.1027880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:33.2548174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcj0g943 2022-08-17T12:45:33.2551436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcj0g943/_remote_module_non_scriptable.py 2022-08-17T12:45:33.5167248Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:33.5177719Z 2022-08-17T12:45:33.5177908Z Running tests... 2022-08-17T12:45:33.5178382Z ---------------------------------------------------------------------- 2022-08-17T12:45:33.7859552Z test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27120 2022-08-17T12:45:33.7887870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27121 2022-08-17T12:45:33.7918624Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27122 2022-08-17T12:45:33.7949465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27123 2022-08-17T12:45:34.8387594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:34.8388229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:34.8391112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:34.8391724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:34.8491954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:34.8492591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:34.8495333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:34.8495950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:34.8681000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:34.8681632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:34.8683148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:34.8683727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:34.8792492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:34.8793220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:34.8796714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:34.8797629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:34.9992586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbobhe18 2022-08-17T12:45:34.9993441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbobhe18/_remote_module_non_scriptable.py 2022-08-17T12:45:35.0097136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxizhu8on 2022-08-17T12:45:35.0098145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxizhu8on/_remote_module_non_scriptable.py 2022-08-17T12:45:35.0287110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpgw9rm1 2022-08-17T12:45:35.0287937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpgw9rm1/_remote_module_non_scriptable.py 2022-08-17T12:45:35.0384354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ux8mt1_ 2022-08-17T12:45:35.0385547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ux8mt1_/_remote_module_non_scriptable.py 2022-08-17T12:45:35.2611486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:35.2719557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:35.2918594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:35.3010536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:36.0011134Z ok (2.483s) 2022-08-17T12:45:36.0011396Z 2022-08-17T12:45:36.0011820Z ---------------------------------------------------------------------- 2022-08-17T12:45:36.0012258Z Ran 1 test in 2.483s 2022-08-17T12:45:36.0012450Z 2022-08-17T12:45:36.0012539Z OK 2022-08-17T12:45:36.0012693Z 2022-08-17T12:45:36.0012846Z Generating XML reports... 2022-08-17T12:45:36.0047579Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124533.xml 2022-08-17T12:45:37.3137793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:37.3138189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:37.3141243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:37.3141667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:37.4668926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp9fk3z8 2022-08-17T12:45:37.4671909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp9fk3z8/_remote_module_non_scriptable.py 2022-08-17T12:45:37.7347930Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:37.7357985Z 2022-08-17T12:45:37.7358266Z Running tests... 2022-08-17T12:45:37.7358917Z ---------------------------------------------------------------------- 2022-08-17T12:45:38.0055884Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27359 2022-08-17T12:45:38.0085245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27360 2022-08-17T12:45:38.0113928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27361 2022-08-17T12:45:38.0152711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27362 2022-08-17T12:45:39.0359609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:39.0360304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:39.0364270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:39.0365214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:39.0384443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:39.0385072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:39.0388094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:39.0388762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:39.0446465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:39.0447084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:39.0450554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:39.0451259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:39.0520660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:39.0521278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:39.0525121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:39.0525809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:39.1985092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw3v486e 2022-08-17T12:45:39.1986651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw3v486e/_remote_module_non_scriptable.py 2022-08-17T12:45:39.1995882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcq3tod3o 2022-08-17T12:45:39.1997625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcq3tod3o/_remote_module_non_scriptable.py 2022-08-17T12:45:39.2047317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixc8o6vk 2022-08-17T12:45:39.2049796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixc8o6vk/_remote_module_non_scriptable.py 2022-08-17T12:45:39.2113957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3m1a12x 2022-08-17T12:45:39.2115464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3m1a12x/_remote_module_non_scriptable.py 2022-08-17T12:45:39.4627427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:39.4628032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:39.4672339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:39.4733100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:40.2214775Z ok (2.485s) 2022-08-17T12:45:40.2215106Z 2022-08-17T12:45:40.2215537Z ---------------------------------------------------------------------- 2022-08-17T12:45:40.2215802Z Ran 1 test in 2.486s 2022-08-17T12:45:40.2215924Z 2022-08-17T12:45:40.2215975Z OK 2022-08-17T12:45:40.2216070Z 2022-08-17T12:45:40.2216167Z Generating XML reports... 2022-08-17T12:45:40.2249374Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124537.xml 2022-08-17T12:45:41.5163304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:41.5163703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:41.5166511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:41.5167095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:41.6697206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r4t2u_5 2022-08-17T12:45:41.6700656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r4t2u_5/_remote_module_non_scriptable.py 2022-08-17T12:45:41.9369318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:41.9380157Z 2022-08-17T12:45:41.9380617Z Running tests... 2022-08-17T12:45:41.9380957Z ---------------------------------------------------------------------- 2022-08-17T12:45:42.2058905Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27586 2022-08-17T12:45:42.2087773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27587 2022-08-17T12:45:42.2117378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27588 2022-08-17T12:45:42.2149260Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27589 2022-08-17T12:45:43.2361621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:43.2362315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:43.2364664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:43.2365345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:43.2770730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:43.2771380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:43.2774467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:43.2775171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:43.3092600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:43.3093289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:43.3094998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:43.3095669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:43.3297828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:43.3298421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:43.3301410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:43.3302147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:43.3930347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_ndpr5f 2022-08-17T12:45:43.3931342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_ndpr5f/_remote_module_non_scriptable.py 2022-08-17T12:45:43.4362892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a5gea9j 2022-08-17T12:45:43.4363717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a5gea9j/_remote_module_non_scriptable.py 2022-08-17T12:45:43.4676766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7elbizw 2022-08-17T12:45:43.4678044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7elbizw/_remote_module_non_scriptable.py 2022-08-17T12:45:43.4888953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe59zopor 2022-08-17T12:45:43.4889790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe59zopor/_remote_module_non_scriptable.py 2022-08-17T12:45:43.6547782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:43.7002692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:43.7319563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:43.7522630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:44.5212221Z ok (2.583s) 2022-08-17T12:45:44.5212453Z 2022-08-17T12:45:44.5212937Z ---------------------------------------------------------------------- 2022-08-17T12:45:44.5213419Z Ran 1 test in 2.583s 2022-08-17T12:45:44.5213623Z 2022-08-17T12:45:44.5213742Z OK 2022-08-17T12:45:44.5213916Z 2022-08-17T12:45:44.5214019Z Generating XML reports... 2022-08-17T12:45:44.5249577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124541.xml 2022-08-17T12:45:45.8269632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:45.8270039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:45.8272749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:45.8273137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:45.9781025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyn3b3fm 2022-08-17T12:45:45.9784371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyn3b3fm/_remote_module_non_scriptable.py 2022-08-17T12:45:46.2424750Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:46.2435629Z 2022-08-17T12:45:46.2435724Z Running tests... 2022-08-17T12:45:46.2436135Z ---------------------------------------------------------------------- 2022-08-17T12:45:46.5114684Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27813 2022-08-17T12:45:46.5144216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27814 2022-08-17T12:45:46.5174622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27815 2022-08-17T12:45:46.5206162Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27816 2022-08-17T12:45:47.5430210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:47.5430902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:47.5434611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:47.5435254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:47.5581463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:47.5582114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:47.5584939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:47.5585687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:47.5625186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:47.5625956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:47.5630567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:47.5631264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:47.5633161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:47.5634035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:47.5638601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:47.5639210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:47.7018100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uwtwqre 2022-08-17T12:45:47.7018942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uwtwqre/_remote_module_non_scriptable.py 2022-08-17T12:45:47.7186528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ljshhhp 2022-08-17T12:45:47.7187289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ljshhhp/_remote_module_non_scriptable.py 2022-08-17T12:45:47.7235779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2ex6zdy 2022-08-17T12:45:47.7237818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2ex6zdy/_remote_module_non_scriptable.py 2022-08-17T12:45:47.7238594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeydt91qb 2022-08-17T12:45:47.7242519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeydt91qb/_remote_module_non_scriptable.py 2022-08-17T12:45:47.9637537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:47.9816447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:47.9860328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:47.9865269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:48.7268983Z ok (2.483s) 2022-08-17T12:45:48.7269221Z 2022-08-17T12:45:48.7269716Z ---------------------------------------------------------------------- 2022-08-17T12:45:48.7270113Z Ran 1 test in 2.483s 2022-08-17T12:45:48.7270305Z 2022-08-17T12:45:48.7270423Z OK 2022-08-17T12:45:48.7270566Z 2022-08-17T12:45:48.7270704Z Generating XML reports... 2022-08-17T12:45:48.7305116Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124546.xml 2022-08-17T12:45:50.0552120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:50.0552808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:50.0555736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:50.0556680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:50.2067347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xj7acno 2022-08-17T12:45:50.2070996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xj7acno/_remote_module_non_scriptable.py 2022-08-17T12:45:50.4702184Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:50.4713696Z 2022-08-17T12:45:50.4714010Z Running tests... 2022-08-17T12:45:50.4714687Z ---------------------------------------------------------------------- 2022-08-17T12:45:50.7362684Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28040 2022-08-17T12:45:50.7390732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28041 2022-08-17T12:45:50.7419226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28042 2022-08-17T12:45:50.7451308Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28043 2022-08-17T12:45:51.7966419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:51.7967316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:51.7969580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:51.7970168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:51.8447314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:51.8447889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:51.8451347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:51.8452059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:51.9220191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:51.9221436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:51.9222557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:51.9223129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:51.9232404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:51.9232960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:51.9236805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:51.9237585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:51.9542714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzc_3jj5 2022-08-17T12:45:52.0028877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzc_3jj5/_remote_module_non_scriptable.py 2022-08-17T12:45:52.0029613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplws5955l 2022-08-17T12:45:52.0030306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplws5955l/_remote_module_non_scriptable.py 2022-08-17T12:45:52.0819804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxxww83z 2022-08-17T12:45:52.0820721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxxww83z/_remote_module_non_scriptable.py 2022-08-17T12:45:52.0838776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uw1gdqx 2022-08-17T12:45:52.0841887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uw1gdqx/_remote_module_non_scriptable.py 2022-08-17T12:45:52.2184963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:52.2648313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:52.3407669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:52.3480341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:53.0512323Z ok (2.580s) 2022-08-17T12:45:53.0512563Z 2022-08-17T12:45:53.0512972Z ---------------------------------------------------------------------- 2022-08-17T12:45:53.0513241Z Ran 1 test in 2.580s 2022-08-17T12:45:53.0513349Z 2022-08-17T12:45:53.0513441Z OK 2022-08-17T12:45:53.0513582Z 2022-08-17T12:45:53.0513680Z Generating XML reports... 2022-08-17T12:45:53.0549747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124550.xml 2022-08-17T12:45:54.3298826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:54.3299222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:54.3312118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:54.3312804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:54.4841769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd26xzqm_ 2022-08-17T12:45:54.4844706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd26xzqm_/_remote_module_non_scriptable.py 2022-08-17T12:45:54.7528883Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:54.7539632Z 2022-08-17T12:45:54.7539762Z Running tests... 2022-08-17T12:45:54.7540487Z ---------------------------------------------------------------------- 2022-08-17T12:45:55.0207115Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28267 2022-08-17T12:45:55.0235032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28268 2022-08-17T12:45:55.0263587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28269 2022-08-17T12:45:55.0296502Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28270 2022-08-17T12:45:56.0489638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:56.0490224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:56.0492879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:56.0493483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:56.0504368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:56.0505025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:56.0508950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:56.0516218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:56.0517079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:56.0517737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:56.0520878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:56.0521598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:56.1149734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:56.1150684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:56.1153362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:56.1153975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:56.2079615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfrxna0m 2022-08-17T12:45:56.2080429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfrxna0m/_remote_module_non_scriptable.py 2022-08-17T12:45:56.2090507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszjnkmhn 2022-08-17T12:45:56.2092255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszjnkmhn/_remote_module_non_scriptable.py 2022-08-17T12:45:56.2097493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ikxn1ei 2022-08-17T12:45:56.2100020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ikxn1ei/_remote_module_non_scriptable.py 2022-08-17T12:45:56.2720797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p_3djdw 2022-08-17T12:45:56.2722159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p_3djdw/_remote_module_non_scriptable.py 2022-08-17T12:45:56.4678366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:45:56.4680935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:45:56.4696449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:45:56.5351708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:45:57.3359100Z ok (2.582s) 2022-08-17T12:45:57.3359301Z 2022-08-17T12:45:57.3359776Z ---------------------------------------------------------------------- 2022-08-17T12:45:57.3360301Z Ran 1 test in 2.582s 2022-08-17T12:45:57.3360472Z 2022-08-17T12:45:57.3360536Z OK 2022-08-17T12:45:57.3360633Z 2022-08-17T12:45:57.3360729Z Generating XML reports... 2022-08-17T12:45:57.3394053Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124554.xml 2022-08-17T12:45:58.6416173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:45:58.6416557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:45:58.6418283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:45:58.6418662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:45:58.7938684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxijb83ch 2022-08-17T12:45:58.7941923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxijb83ch/_remote_module_non_scriptable.py 2022-08-17T12:45:59.0597530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:45:59.0608468Z 2022-08-17T12:45:59.0608895Z Running tests... 2022-08-17T12:45:59.0609348Z ---------------------------------------------------------------------- 2022-08-17T12:45:59.3248172Z test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28490 2022-08-17T12:45:59.3276361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28491 2022-08-17T12:45:59.3305597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28492 2022-08-17T12:45:59.3339443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28493 2022-08-17T12:46:00.3635464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:00.3636463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:00.3638172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:00.3638815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:00.3926317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:00.3926948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:00.3929510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:00.3930122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:00.4062408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:00.4063138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:00.4064357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:00.4066738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:00.4067978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:00.4068643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:00.4070642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:00.4071331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:00.5212915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2hiq3zq 2022-08-17T12:46:00.5213903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2hiq3zq/_remote_module_non_scriptable.py 2022-08-17T12:46:00.5519703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9efbz1o 2022-08-17T12:46:00.5520655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9efbz1o/_remote_module_non_scriptable.py 2022-08-17T12:46:00.5649978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp226y9x7o 2022-08-17T12:46:00.5650791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp226y9x7o/_remote_module_non_scriptable.py 2022-08-17T12:46:00.5668284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkbswzxb 2022-08-17T12:46:00.5670303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkbswzxb/_remote_module_non_scriptable.py 2022-08-17T12:46:00.7830573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:00.8141696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:00.8247857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:00.8318106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:01.5400177Z ok (2.479s) 2022-08-17T12:46:01.5400417Z 2022-08-17T12:46:01.5400763Z ---------------------------------------------------------------------- 2022-08-17T12:46:01.5401028Z Ran 1 test in 2.479s 2022-08-17T12:46:01.5401151Z 2022-08-17T12:46:01.5401215Z OK 2022-08-17T12:46:01.5401313Z 2022-08-17T12:46:01.5401408Z Generating XML reports... 2022-08-17T12:46:01.5435707Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124559.xml 2022-08-17T12:46:02.8630288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:02.8630922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:02.8632732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:02.8633370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:03.0126668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1sta2xer 2022-08-17T12:46:03.0130040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1sta2xer/_remote_module_non_scriptable.py 2022-08-17T12:46:03.2759268Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:03.2770475Z 2022-08-17T12:46:03.2770938Z Running tests... 2022-08-17T12:46:03.2771358Z ---------------------------------------------------------------------- 2022-08-17T12:46:03.5374217Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28713 2022-08-17T12:46:03.5402153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28714 2022-08-17T12:46:03.5430621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28715 2022-08-17T12:46:03.5462075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28716 2022-08-17T12:46:04.5860618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:04.5861304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:04.5863709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:04.5864353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:04.5881415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:04.5882062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:04.5886416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:04.5887053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:04.6156546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:04.6157206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:04.6160292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:04.6160985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:04.6329607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:04.6330310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:04.6333481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:04.6334114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:04.7443743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf2vzho8 2022-08-17T12:46:04.7446066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf2vzho8/_remote_module_non_scriptable.py 2022-08-17T12:46:04.7471979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3k2k2vo 2022-08-17T12:46:04.7473932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3k2k2vo/_remote_module_non_scriptable.py 2022-08-17T12:46:04.7753577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uh1kf80 2022-08-17T12:46:04.7754430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uh1kf80/_remote_module_non_scriptable.py 2022-08-17T12:46:04.7907030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mokxyci 2022-08-17T12:46:04.7908454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mokxyci/_remote_module_non_scriptable.py 2022-08-17T12:46:05.0029836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:05.0105028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:05.0350747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:05.0508559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:05.7525129Z ok (2.475s) 2022-08-17T12:46:05.7525384Z 2022-08-17T12:46:05.7526458Z ---------------------------------------------------------------------- 2022-08-17T12:46:05.7526847Z Ran 1 test in 2.475s 2022-08-17T12:46:05.7527033Z 2022-08-17T12:46:05.7527883Z OK 2022-08-17T12:46:05.7528465Z 2022-08-17T12:46:05.7528632Z Generating XML reports... 2022-08-17T12:46:05.7564561Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124603.xml 2022-08-17T12:46:07.0585248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:07.0585971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:07.0588728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:07.0589222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:07.2188676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhjyyoyd 2022-08-17T12:46:07.2191236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhjyyoyd/_remote_module_non_scriptable.py 2022-08-17T12:46:07.4874647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:07.4886113Z 2022-08-17T12:46:07.4886220Z Running tests... 2022-08-17T12:46:07.4886802Z ---------------------------------------------------------------------- 2022-08-17T12:46:07.7573793Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28936 2022-08-17T12:46:07.7604709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28937 2022-08-17T12:46:07.7633211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28938 2022-08-17T12:46:07.7663564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28939 2022-08-17T12:46:08.7593269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:08.7593969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:08.7597186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:08.7597847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:08.7893579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:08.7894281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:08.7897295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:08.7897942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:08.8104853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:08.8105487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:08.8108559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:08.8109464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:08.8264938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:08.8265638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:08.8268688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:08.8269327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:08.9186840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7it3s1g 2022-08-17T12:46:08.9187656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7it3s1g/_remote_module_non_scriptable.py 2022-08-17T12:46:08.9505605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvivx4hj 2022-08-17T12:46:08.9506408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvivx4hj/_remote_module_non_scriptable.py 2022-08-17T12:46:08.9718060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfh4k_9q 2022-08-17T12:46:08.9718848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfh4k_9q/_remote_module_non_scriptable.py 2022-08-17T12:46:08.9861178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77jbvoqi 2022-08-17T12:46:08.9862058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77jbvoqi/_remote_module_non_scriptable.py 2022-08-17T12:46:09.1809263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:09.2172802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:09.2350042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:09.2512147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:09.9725269Z ok (2.484s) 2022-08-17T12:46:09.9726034Z 2022-08-17T12:46:09.9726585Z ---------------------------------------------------------------------- 2022-08-17T12:46:09.9726856Z Ran 1 test in 2.484s 2022-08-17T12:46:09.9726963Z 2022-08-17T12:46:09.9727031Z OK 2022-08-17T12:46:09.9727129Z 2022-08-17T12:46:09.9727239Z Generating XML reports... 2022-08-17T12:46:09.9759853Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124607.xml 2022-08-17T12:46:11.2958370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:11.2959093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:11.2962438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:11.2963053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:11.4502018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie0w_czx 2022-08-17T12:46:11.4505493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie0w_czx/_remote_module_non_scriptable.py 2022-08-17T12:46:11.7127681Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:11.7138890Z 2022-08-17T12:46:11.7139369Z Running tests... 2022-08-17T12:46:11.7139794Z ---------------------------------------------------------------------- 2022-08-17T12:46:11.9784353Z test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29159 2022-08-17T12:46:11.9813904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29160 2022-08-17T12:46:11.9843111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29161 2022-08-17T12:46:11.9874758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29162 2022-08-17T12:46:13.0130186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:13.0130788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:13.0134594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:13.0135254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:13.0178228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:13.0178857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:13.0182894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:13.0183519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:13.0185741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:13.0186291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:13.0191329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:13.0338432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:13.0339402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:13.0340002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:13.0341487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:13.0342125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:13.1741729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bylx8cp 2022-08-17T12:46:13.1743111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bylx8cp/_remote_module_non_scriptable.py 2022-08-17T12:46:13.1790449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7c9nt6m8 2022-08-17T12:46:13.1791316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7c9nt6m8/_remote_module_non_scriptable.py 2022-08-17T12:46:13.1808536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq9lqpa7 2022-08-17T12:46:13.1810962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq9lqpa7/_remote_module_non_scriptable.py 2022-08-17T12:46:13.1946480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp0vtxkx 2022-08-17T12:46:13.1947485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp0vtxkx/_remote_module_non_scriptable.py 2022-08-17T12:46:13.4347134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:13.4446695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:13.4474292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:13.4573986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:14.1935851Z ok (2.479s) 2022-08-17T12:46:14.1936040Z 2022-08-17T12:46:14.1936553Z ---------------------------------------------------------------------- 2022-08-17T12:46:14.1936972Z Ran 1 test in 2.480s 2022-08-17T12:46:14.1937097Z 2022-08-17T12:46:14.1937161Z OK 2022-08-17T12:46:14.1937261Z 2022-08-17T12:46:14.1937360Z Generating XML reports... 2022-08-17T12:46:14.1970142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124611.xml 2022-08-17T12:46:15.5170379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:15.5171315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:15.5173650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:15.5174248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:15.6706893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3oaja3_ 2022-08-17T12:46:15.6710508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3oaja3_/_remote_module_non_scriptable.py 2022-08-17T12:46:15.9343720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:15.9354832Z 2022-08-17T12:46:15.9355140Z Running tests... 2022-08-17T12:46:15.9355801Z ---------------------------------------------------------------------- 2022-08-17T12:46:16.1986492Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29382 2022-08-17T12:46:16.2014947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29383 2022-08-17T12:46:16.2042726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29384 2022-08-17T12:46:16.2074048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29385 2022-08-17T12:46:17.1982097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:17.1982748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:17.1985023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:17.1985755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:17.2247328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:17.2247993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:17.2250450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:17.2251084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:17.2294631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:17.2295253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:17.2298821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:17.2299448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:17.2406325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:17.2407062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:17.2410231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:17.2410938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:17.3576740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2jbbhve 2022-08-17T12:46:17.3577607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2jbbhve/_remote_module_non_scriptable.py 2022-08-17T12:46:17.3848261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv2ks193 2022-08-17T12:46:17.3849494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv2ks193/_remote_module_non_scriptable.py 2022-08-17T12:46:17.3894475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9doo9zq9 2022-08-17T12:46:17.3895980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9doo9zq9/_remote_module_non_scriptable.py 2022-08-17T12:46:17.3984892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p5vxaj5 2022-08-17T12:46:17.3986396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p5vxaj5/_remote_module_non_scriptable.py 2022-08-17T12:46:17.6366866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:17.6453953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:17.6502275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:17.6664157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:18.4136390Z ok (2.478s) 2022-08-17T12:46:18.4136679Z 2022-08-17T12:46:18.4137232Z ---------------------------------------------------------------------- 2022-08-17T12:46:18.4137554Z Ran 1 test in 2.478s 2022-08-17T12:46:18.4137693Z 2022-08-17T12:46:18.4137757Z OK 2022-08-17T12:46:18.4137840Z 2022-08-17T12:46:18.4137940Z Generating XML reports... 2022-08-17T12:46:18.4172045Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124615.xml 2022-08-17T12:46:19.7274788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:19.7275234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:19.7276939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:19.7277353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:19.8812593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpste9j87j 2022-08-17T12:46:19.8815870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpste9j87j/_remote_module_non_scriptable.py 2022-08-17T12:46:20.1488645Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:20.1499834Z 2022-08-17T12:46:20.1499918Z Running tests... 2022-08-17T12:46:20.1500343Z ---------------------------------------------------------------------- 2022-08-17T12:46:20.4148002Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29605 2022-08-17T12:46:20.4177344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29606 2022-08-17T12:46:20.4206221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29607 2022-08-17T12:46:20.4237038Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29608 2022-08-17T12:46:21.5095223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:21.5095899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:21.5098893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:21.5099560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:21.5263940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:21.5264568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:21.5265325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:21.5266020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:21.5267049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:21.5267634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:21.5271791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:21.5272475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:21.5285827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:21.5286674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:21.5290859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:21.5291545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:21.6707759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgrt7ivu 2022-08-17T12:46:21.6708617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgrt7ivu/_remote_module_non_scriptable.py 2022-08-17T12:46:21.6859293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_2yxjqy 2022-08-17T12:46:21.6860107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_2yxjqy/_remote_module_non_scriptable.py 2022-08-17T12:46:21.6877538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vkuumoc 2022-08-17T12:46:21.6879487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vkuumoc/_remote_module_non_scriptable.py 2022-08-17T12:46:21.6886006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqftxrx4s 2022-08-17T12:46:21.6889152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqftxrx4s/_remote_module_non_scriptable.py 2022-08-17T12:46:21.9506559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:21.9521121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:21.9524655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:21.9606827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:22.7300400Z ok (2.580s) 2022-08-17T12:46:22.7300678Z 2022-08-17T12:46:22.7301228Z ---------------------------------------------------------------------- 2022-08-17T12:46:22.7301635Z Ran 1 test in 2.580s 2022-08-17T12:46:22.7301766Z 2022-08-17T12:46:22.7301830Z OK 2022-08-17T12:46:22.7301927Z 2022-08-17T12:46:22.7302022Z Generating XML reports... 2022-08-17T12:46:22.7334414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124620.xml 2022-08-17T12:46:24.0336706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:24.0337149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:24.0339410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:24.0339960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:24.1838743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopx29trj 2022-08-17T12:46:24.1842335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopx29trj/_remote_module_non_scriptable.py 2022-08-17T12:46:24.4469373Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:24.4480004Z 2022-08-17T12:46:24.4480125Z Running tests... 2022-08-17T12:46:24.4480762Z ---------------------------------------------------------------------- 2022-08-17T12:46:24.7120877Z test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29832 2022-08-17T12:46:24.7149142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29833 2022-08-17T12:46:24.7177655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29834 2022-08-17T12:46:24.7208629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29835 2022-08-17T12:46:25.7702837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:25.7703467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:25.7706468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:25.7707149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:25.7797257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:25.7797890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:25.7801564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:25.7802280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:25.7848398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:25.7849370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:25.7852546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:25.7853208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:25.8132189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:25.8132893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:25.8135864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:25.8136536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:25.9412858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_b_ckef_ 2022-08-17T12:46:25.9413697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_b_ckef_/_remote_module_non_scriptable.py 2022-08-17T12:46:25.9429338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5p3icyyt 2022-08-17T12:46:25.9431354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5p3icyyt/_remote_module_non_scriptable.py 2022-08-17T12:46:25.9433856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl9rkca0 2022-08-17T12:46:25.9437136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl9rkca0/_remote_module_non_scriptable.py 2022-08-17T12:46:25.9689663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmuzu_gm 2022-08-17T12:46:25.9690763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmuzu_gm/_remote_module_non_scriptable.py 2022-08-17T12:46:26.2017443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:26.2035042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:26.2110503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:26.2271700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:26.9269169Z ok (2.479s) 2022-08-17T12:46:26.9269473Z 2022-08-17T12:46:26.9269863Z ---------------------------------------------------------------------- 2022-08-17T12:46:26.9270131Z Ran 1 test in 2.479s 2022-08-17T12:46:26.9270254Z 2022-08-17T12:46:26.9270318Z OK 2022-08-17T12:46:26.9270414Z 2022-08-17T12:46:26.9270511Z Generating XML reports... 2022-08-17T12:46:26.9306010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124624.xml 2022-08-17T12:46:28.2232431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:28.2233017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:28.2236260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:28.2236929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:28.3745241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsne6hp9l 2022-08-17T12:46:28.3748207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsne6hp9l/_remote_module_non_scriptable.py 2022-08-17T12:46:28.6368677Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:28.6379622Z 2022-08-17T12:46:28.6380068Z Running tests... 2022-08-17T12:46:28.6380618Z ---------------------------------------------------------------------- 2022-08-17T12:46:28.9001299Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30055 2022-08-17T12:46:28.9030857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30056 2022-08-17T12:46:28.9060018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30057 2022-08-17T12:46:28.9092507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30058 2022-08-17T12:46:29.9490614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:29.9491307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:29.9493720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:29.9494375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:29.9608571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:29.9609260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:29.9613125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:29.9613775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:29.9642363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:29.9642967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:29.9646833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:29.9647486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:29.9707292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:29.9708320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:29.9711917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:29.9712593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:30.1082274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ymm7kz5 2022-08-17T12:46:30.1083066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ymm7kz5/_remote_module_non_scriptable.py 2022-08-17T12:46:30.1200927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68rdqmeu 2022-08-17T12:46:30.1201703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68rdqmeu/_remote_module_non_scriptable.py 2022-08-17T12:46:30.1224578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqvsmgyw 2022-08-17T12:46:30.1226116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqvsmgyw/_remote_module_non_scriptable.py 2022-08-17T12:46:30.1519456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynjmz3k2 2022-08-17T12:46:30.1520283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynjmz3k2/_remote_module_non_scriptable.py 2022-08-17T12:46:30.3674833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:30.3787298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:30.3800726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:30.4142005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:31.0151104Z ok (2.377s) 2022-08-17T12:46:31.0151358Z 2022-08-17T12:46:31.0151779Z ---------------------------------------------------------------------- 2022-08-17T12:46:31.0152051Z Ran 1 test in 2.377s 2022-08-17T12:46:31.0152160Z 2022-08-17T12:46:31.0152223Z OK 2022-08-17T12:46:31.0152320Z 2022-08-17T12:46:31.0152416Z Generating XML reports... 2022-08-17T12:46:31.0186428Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124628.xml 2022-08-17T12:46:32.3259512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:32.3259909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:32.3262656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:32.3263025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:32.4818702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuba8k8fn 2022-08-17T12:46:32.4822849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuba8k8fn/_remote_module_non_scriptable.py 2022-08-17T12:46:32.7506372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:32.7517424Z 2022-08-17T12:46:32.7517574Z Running tests... 2022-08-17T12:46:32.7518053Z ---------------------------------------------------------------------- 2022-08-17T12:46:33.0193161Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30278 2022-08-17T12:46:33.0223849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30279 2022-08-17T12:46:33.0253624Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30280 2022-08-17T12:46:33.0288321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30281 2022-08-17T12:46:34.0760112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:34.0760737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:34.0762238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:34.0762887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:34.0763667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:34.0764287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:34.0769016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:34.0769662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:34.0899404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:34.0900325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:34.0902464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:34.0903134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:34.1120876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:34.1121563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:34.1122673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:34.1123297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:34.2380497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98ed41ii 2022-08-17T12:46:34.2381636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98ed41ii/_remote_module_non_scriptable.py 2022-08-17T12:46:34.2386748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu94m4j4j 2022-08-17T12:46:34.2389594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu94m4j4j/_remote_module_non_scriptable.py 2022-08-17T12:46:34.2497173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmp8enm_ 2022-08-17T12:46:34.2498244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmp8enm_/_remote_module_non_scriptable.py 2022-08-17T12:46:34.2807708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftjkijc8 2022-08-17T12:46:34.2808794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftjkijc8/_remote_module_non_scriptable.py 2022-08-17T12:46:34.5023462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:34.5065901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:34.5172171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:34.5527304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:34.7733317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:46:34.7734260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:46:34.7832249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:46:34.7834023Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:46:34.7835293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:46:34.7836619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:46:34.7837666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:46:34.7838678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:46:35.2347711Z ok (2.483s) 2022-08-17T12:46:35.2347945Z 2022-08-17T12:46:35.2348481Z ---------------------------------------------------------------------- 2022-08-17T12:46:35.2348942Z Ran 1 test in 2.483s 2022-08-17T12:46:35.2349091Z 2022-08-17T12:46:35.2349156Z OK 2022-08-17T12:46:35.2349253Z 2022-08-17T12:46:35.2349339Z Generating XML reports... 2022-08-17T12:46:35.2382937Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124632.xml 2022-08-17T12:46:36.5370445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:36.5371090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:36.5373568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:36.5373956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:36.6902798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei8nfqfz 2022-08-17T12:46:36.6906429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei8nfqfz/_remote_module_non_scriptable.py 2022-08-17T12:46:36.9611484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:36.9622703Z 2022-08-17T12:46:36.9622891Z Running tests... 2022-08-17T12:46:36.9623260Z ---------------------------------------------------------------------- 2022-08-17T12:46:37.2279405Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30513 2022-08-17T12:46:37.2309247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30514 2022-08-17T12:46:37.2339176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30515 2022-08-17T12:46:37.2370549Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30516 2022-08-17T12:46:38.3262448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:38.3263126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:38.3266228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:38.3266879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:38.3542272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:38.3542961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:38.3546091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:38.3546751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:38.3574091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:38.3574750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:38.3578645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:38.3579290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:38.3843853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:38.3844497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:38.3848108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:38.3848796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:38.4896846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h1f1jfs 2022-08-17T12:46:38.4898172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h1f1jfs/_remote_module_non_scriptable.py 2022-08-17T12:46:38.5191239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t3n0vd8 2022-08-17T12:46:38.5192000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t3n0vd8/_remote_module_non_scriptable.py 2022-08-17T12:46:38.5335406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjfops74 2022-08-17T12:46:38.5337126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjfops74/_remote_module_non_scriptable.py 2022-08-17T12:46:38.5433071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5blsv02m 2022-08-17T12:46:38.5434886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5blsv02m/_remote_module_non_scriptable.py 2022-08-17T12:46:38.7508898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:38.7829599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:38.7908027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:38.8040794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:39.4432123Z ok (2.481s) 2022-08-17T12:46:39.4432290Z 2022-08-17T12:46:39.4432620Z ---------------------------------------------------------------------- 2022-08-17T12:46:39.4432949Z Ran 1 test in 2.481s 2022-08-17T12:46:39.4433071Z 2022-08-17T12:46:39.4433136Z OK 2022-08-17T12:46:39.4433231Z 2022-08-17T12:46:39.4433328Z Generating XML reports... 2022-08-17T12:46:39.4466348Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124636.xml 2022-08-17T12:46:40.7580611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:40.7581028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:40.7583694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:40.7584079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:40.9128613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xqmhw6w 2022-08-17T12:46:40.9132224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xqmhw6w/_remote_module_non_scriptable.py 2022-08-17T12:46:41.1801860Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:41.1813475Z 2022-08-17T12:46:41.1813567Z Running tests... 2022-08-17T12:46:41.1814414Z ---------------------------------------------------------------------- 2022-08-17T12:46:41.4472279Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30736 2022-08-17T12:46:41.4500081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30737 2022-08-17T12:46:41.4528120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30738 2022-08-17T12:46:41.4559702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30739 2022-08-17T12:46:42.5160582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:42.5161264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:42.5162791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:42.5163442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:42.5613833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:42.5614497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:42.5617524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:42.5618141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:42.5685175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:42.5685707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:42.5691284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:42.5691956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:42.5767026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:42.5767655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:42.5771979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:42.5772631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:42.6744717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptws6whww 2022-08-17T12:46:42.6745538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptws6whww/_remote_module_non_scriptable.py 2022-08-17T12:46:42.7201983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvjh7bsy 2022-08-17T12:46:42.7202836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvjh7bsy/_remote_module_non_scriptable.py 2022-08-17T12:46:42.7376847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01ikvy5k 2022-08-17T12:46:42.7378275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01ikvy5k/_remote_module_non_scriptable.py 2022-08-17T12:46:42.7435264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2apa2crs 2022-08-17T12:46:42.7436710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2apa2crs/_remote_module_non_scriptable.py 2022-08-17T12:46:42.9351625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:42.9792294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:42.9946728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:43.0065206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:43.7623203Z ok (2.581s) 2022-08-17T12:46:43.7623497Z 2022-08-17T12:46:43.7624040Z ---------------------------------------------------------------------- 2022-08-17T12:46:43.7624297Z Ran 1 test in 2.581s 2022-08-17T12:46:43.7624419Z 2022-08-17T12:46:43.7624484Z OK 2022-08-17T12:46:43.7624582Z 2022-08-17T12:46:43.7624679Z Generating XML reports... 2022-08-17T12:46:43.7658455Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124641.xml 2022-08-17T12:46:45.0640692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:45.0641384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:45.0644466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:45.0645104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:45.2169902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvld00ju 2022-08-17T12:46:45.2174133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvld00ju/_remote_module_non_scriptable.py 2022-08-17T12:46:45.4816384Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:45.4827351Z 2022-08-17T12:46:45.4828027Z Running tests... 2022-08-17T12:46:45.4828695Z ---------------------------------------------------------------------- 2022-08-17T12:46:45.7485466Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30959 2022-08-17T12:46:45.7515206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30960 2022-08-17T12:46:45.7546781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30961 2022-08-17T12:46:45.7578691Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30962 2022-08-17T12:46:46.8687300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:46.8688001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:46.8690803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:46.8691492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:46.8750717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:46.8751287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:46.8754790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:46.8755473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:46.8796004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:46.8796907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:46.8800507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:46.8801109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:46.9600343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:46.9600976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:46.9604275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:46.9604953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:47.0290188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uvr6csz 2022-08-17T12:46:47.0290952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uvr6csz/_remote_module_non_scriptable.py 2022-08-17T12:46:47.0361465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_rpuwbn 2022-08-17T12:46:47.0363127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_rpuwbn/_remote_module_non_scriptable.py 2022-08-17T12:46:47.0404278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnfx4tjx 2022-08-17T12:46:47.0406063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnfx4tjx/_remote_module_non_scriptable.py 2022-08-17T12:46:47.1186412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnd6drdv 2022-08-17T12:46:47.1187245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnd6drdv/_remote_module_non_scriptable.py 2022-08-17T12:46:47.2911812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:47.3004647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:47.3037922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:47.3826482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:48.1640448Z ok (2.681s) 2022-08-17T12:46:48.1640603Z 2022-08-17T12:46:48.1641031Z ---------------------------------------------------------------------- 2022-08-17T12:46:48.1641298Z Ran 1 test in 2.681s 2022-08-17T12:46:48.1641422Z 2022-08-17T12:46:48.1641488Z OK 2022-08-17T12:46:48.1641583Z 2022-08-17T12:46:48.1641665Z Generating XML reports... 2022-08-17T12:46:48.1678573Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124645.xml 2022-08-17T12:46:49.4722367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:49.4722852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:49.4724906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:49.4725343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:49.6252657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8skmnt3 2022-08-17T12:46:49.6256133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8skmnt3/_remote_module_non_scriptable.py 2022-08-17T12:46:49.8897858Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:49.8908842Z 2022-08-17T12:46:49.8909015Z Running tests... 2022-08-17T12:46:49.8909518Z ---------------------------------------------------------------------- 2022-08-17T12:46:50.1569771Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31182 2022-08-17T12:46:50.1599395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31183 2022-08-17T12:46:50.1628256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31184 2022-08-17T12:46:50.1658612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31185 2022-08-17T12:46:51.2256771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:51.2257294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:51.2260034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:51.2260695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:51.2335563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:51.2336209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:51.2339007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:51.2339676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:51.2726571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:51.2727135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:51.2730420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:51.2730973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:51.2976804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:51.2977531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:51.2980709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:51.2981441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:51.3846591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_484qfo 2022-08-17T12:46:51.3847429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_484qfo/_remote_module_non_scriptable.py 2022-08-17T12:46:51.3925324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwff230f 2022-08-17T12:46:51.3927010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwff230f/_remote_module_non_scriptable.py 2022-08-17T12:46:51.4307960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u0pb9d2 2022-08-17T12:46:51.4309146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u0pb9d2/_remote_module_non_scriptable.py 2022-08-17T12:46:51.4560280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyy0lge0 2022-08-17T12:46:51.4561275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyy0lge0/_remote_module_non_scriptable.py 2022-08-17T12:46:51.6433243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:51.6532239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:51.6892822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:51.7171274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:51.9388236Z 12:46:51 Start training 2022-08-17T12:46:51.9433229Z 12:46:51 worker1 processing one batch 2022-08-17T12:46:51.9438560Z 12:46:51 worker2 processing one batch 2022-08-17T12:46:51.9445516Z 12:46:51 worker1 reporting grads 2022-08-17T12:46:51.9451876Z 12:46:51 worker2 reporting grads 2022-08-17T12:46:51.9453892Z 12:46:51 worker3 processing one batch 2022-08-17T12:46:51.9456937Z 12:46:51 PS got 0/3 updates 2022-08-17T12:46:51.9466692Z 12:46:51 PS got 1/3 updates 2022-08-17T12:46:51.9472953Z 12:46:51 worker3 reporting grads 2022-08-17T12:46:51.9480214Z 12:46:51 PS got 2/3 updates 2022-08-17T12:46:51.9490274Z 12:46:51 PS updated model 2022-08-17T12:46:51.9493787Z 12:46:51 worker1 got updated model 2022-08-17T12:46:51.9494307Z 12:46:51 worker1 processing one batch 2022-08-17T12:46:51.9498802Z 12:46:51 worker2 got updated model 2022-08-17T12:46:51.9499754Z 12:46:51 worker1 reporting grads 2022-08-17T12:46:51.9500449Z 12:46:51 worker2 processing one batch 2022-08-17T12:46:51.9508259Z 12:46:51 worker3 got updated model 2022-08-17T12:46:51.9508970Z 12:46:51 PS got 0/3 updates 2022-08-17T12:46:51.9509409Z 12:46:51 worker2 reporting grads 2022-08-17T12:46:51.9509865Z 12:46:51 worker3 processing one batch 2022-08-17T12:46:51.9514973Z 12:46:51 worker3 reporting grads 2022-08-17T12:46:51.9520695Z 12:46:51 PS got 1/3 updates 2022-08-17T12:46:51.9526642Z 12:46:51 PS got 2/3 updates 2022-08-17T12:46:51.9536834Z 12:46:51 PS updated model 2022-08-17T12:46:51.9543722Z 12:46:51 worker2 got updated model 2022-08-17T12:46:51.9544362Z 12:46:51 worker2 processing one batch 2022-08-17T12:46:51.9544835Z 12:46:51 worker1 got updated model 2022-08-17T12:46:51.9545311Z 12:46:51 worker3 got updated model 2022-08-17T12:46:51.9546312Z 12:46:51 worker3 processing one batch 2022-08-17T12:46:51.9548058Z 12:46:51 worker2 reporting grads 2022-08-17T12:46:51.9548738Z 12:46:51 worker3 reporting grads 2022-08-17T12:46:51.9552786Z 12:46:51 worker1 processing one batch 2022-08-17T12:46:51.9557189Z 12:46:51 PS got 0/3 updates 2022-08-17T12:46:51.9561299Z 12:46:51 PS got 1/3 updates 2022-08-17T12:46:51.9566479Z 12:46:51 worker1 reporting grads 2022-08-17T12:46:51.9577180Z 12:46:51 PS got 2/3 updates 2022-08-17T12:46:51.9586363Z 12:46:51 PS updated model 2022-08-17T12:46:51.9590367Z 12:46:51 worker3 got updated model 2022-08-17T12:46:51.9590853Z 12:46:51 worker3 processing one batch 2022-08-17T12:46:51.9594312Z 12:46:51 worker3 reporting grads 2022-08-17T12:46:51.9601320Z 12:46:51 worker1 got updated model 2022-08-17T12:46:51.9601837Z 12:46:51 worker2 got updated model 2022-08-17T12:46:51.9602333Z 12:46:51 worker2 processing one batch 2022-08-17T12:46:51.9602823Z 12:46:51 worker1 processing one batch 2022-08-17T12:46:51.9608003Z 12:46:51 PS got 0/3 updates 2022-08-17T12:46:51.9609010Z 12:46:51 worker1 reporting grads 2022-08-17T12:46:51.9609593Z 12:46:51 worker2 reporting grads 2022-08-17T12:46:51.9618386Z 12:46:51 PS got 1/3 updates 2022-08-17T12:46:51.9619822Z 12:46:51 PS got 2/3 updates 2022-08-17T12:46:51.9625151Z 12:46:51 PS updated model 2022-08-17T12:46:51.9628453Z 12:46:51 worker3 got updated model 2022-08-17T12:46:51.9630108Z 12:46:51 worker2 got updated model 2022-08-17T12:46:51.9632602Z 12:46:51 worker1 got updated model 2022-08-17T12:46:51.9636474Z 12:46:51 Finish training 2022-08-17T12:46:51.9636969Z 12:46:51 Time spent training: 0.025049309999758407s 2022-08-17T12:46:52.3720050Z ok (2.481s) 2022-08-17T12:46:52.3720313Z 2022-08-17T12:46:52.3720845Z ---------------------------------------------------------------------- 2022-08-17T12:46:52.3721215Z Ran 1 test in 2.481s 2022-08-17T12:46:52.3721335Z 2022-08-17T12:46:52.3721398Z OK 2022-08-17T12:46:52.3721492Z 2022-08-17T12:46:52.3721587Z Generating XML reports... 2022-08-17T12:46:52.3754883Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220817124649.xml 2022-08-17T12:46:53.6726759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:53.6727177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:53.6729540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:53.6730023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:53.8249973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvc1lsfom 2022-08-17T12:46:53.8252977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvc1lsfom/_remote_module_non_scriptable.py 2022-08-17T12:46:54.0895423Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:54.0905565Z 2022-08-17T12:46:54.0905775Z Running tests... 2022-08-17T12:46:54.0906377Z ---------------------------------------------------------------------- 2022-08-17T12:46:54.3569047Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31405 2022-08-17T12:46:54.3597168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31406 2022-08-17T12:46:54.3626506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31407 2022-08-17T12:46:54.3659064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31408 2022-08-17T12:46:55.4414005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:55.4414575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:55.4417420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:55.4418006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:55.4418801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:55.4419421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:55.4422886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:55.4423570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:55.4713117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:55.4713738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:55.4715133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:55.4715829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:55.4981767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:55.4982351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:55.4986037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:55.4986627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:55.6022158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2vouals 2022-08-17T12:46:55.6022999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2vouals/_remote_module_non_scriptable.py 2022-08-17T12:46:55.6030235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2483tgk 2022-08-17T12:46:55.6032699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2483tgk/_remote_module_non_scriptable.py 2022-08-17T12:46:55.6296294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4isbqj84 2022-08-17T12:46:55.6297799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4isbqj84/_remote_module_non_scriptable.py 2022-08-17T12:46:55.6573700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yzrd7dz 2022-08-17T12:46:55.6575033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yzrd7dz/_remote_module_non_scriptable.py 2022-08-17T12:46:55.8676863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:46:55.8677865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:46:55.8933938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:46:55.9221569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:46:56.3801597Z Solved! Running reward is now 489.8051609821906! 2022-08-17T12:46:56.7722180Z ok (2.681s) 2022-08-17T12:46:56.7722459Z 2022-08-17T12:46:56.7722833Z ---------------------------------------------------------------------- 2022-08-17T12:46:56.7723114Z Ran 1 test in 2.682s 2022-08-17T12:46:56.7723259Z 2022-08-17T12:46:56.7723322Z OK 2022-08-17T12:46:56.7723416Z 2022-08-17T12:46:56.7723511Z Generating XML reports... 2022-08-17T12:46:56.7756769Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220817124654.xml 2022-08-17T12:46:58.0889038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:58.0889510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:58.0892034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:58.0892409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:58.2410740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5_r_mzw 2022-08-17T12:46:58.2414512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5_r_mzw/_remote_module_non_scriptable.py 2022-08-17T12:46:58.5063136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:46:58.5074180Z 2022-08-17T12:46:58.5074595Z Running tests... 2022-08-17T12:46:58.5075026Z ---------------------------------------------------------------------- 2022-08-17T12:46:58.7704624Z test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31631 2022-08-17T12:46:58.7733142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31632 2022-08-17T12:46:59.7248934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:59.7249578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:59.7251885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:59.7252821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:59.7609484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:46:59.7609924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:46:59.7612299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:46:59.7612663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:46:59.8789111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiwkc8gy 2022-08-17T12:46:59.8790021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiwkc8gy/_remote_module_non_scriptable.py 2022-08-17T12:46:59.9145374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpp7a7ip 2022-08-17T12:46:59.9146264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpp7a7ip/_remote_module_non_scriptable.py 2022-08-17T12:47:00.1358043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:00.1689739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:00.3012783Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3014724Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-08-17T12:47:00.3015676Z Traceback (most recent call last): 2022-08-17T12:47:00.3016645Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3017401Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3018429Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module 2022-08-17T12:47:00.3019229Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-08-17T12:47:00.3020568Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-08-17T12:47:00.3021302Z 2022-08-17T12:47:00.3021521Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3025436Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-08-17T12:47:00.3028025Z Traceback (most recent call last): 2022-08-17T12:47:00.3028911Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3029679Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3030388Z File "/tmp/tmpd5_r_mzw/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-08-17T12:47:00.3031005Z module = module_rref.local_value() 2022-08-17T12:47:00.3031910Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T12:47:00.3032879Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T12:47:00.3033673Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3034974Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-08-17T12:47:00.3035838Z Traceback (most recent call last): 2022-08-17T12:47:00.3036705Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3037382Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3038375Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module 2022-08-17T12:47:00.3039192Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-08-17T12:47:00.3040623Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-08-17T12:47:00.3041282Z 2022-08-17T12:47:00.3041293Z 2022-08-17T12:47:00.3041519Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3042839Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-08-17T12:47:00.3043782Z Traceback (most recent call last): 2022-08-17T12:47:00.3044698Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3045474Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3046717Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module 2022-08-17T12:47:00.3047465Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-08-17T12:47:00.3048880Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-08-17T12:47:00.3049523Z 2022-08-17T12:47:00.3049741Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3053397Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .")\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module\n "Expect `module_cls(*args, **kwargs)` returns an instance of , "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .\n') 2022-08-17T12:47:00.3055874Z Traceback (most recent call last): 2022-08-17T12:47:00.3056773Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3057541Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3058262Z File "/tmp/tmpd5_r_mzw/_remote_module_non_scriptable.py", line 47, in _remote_forward 2022-08-17T12:47:00.3058869Z module = module_rref.local_value() 2022-08-17T12:47:00.3059777Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T12:47:00.3060780Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T12:47:00.3061598Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-08-17T12:47:00.3062928Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of .") 2022-08-17T12:47:00.3063854Z Traceback (most recent call last): 2022-08-17T12:47:00.3064761Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:47:00.3065520Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:47:00.3066620Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module 2022-08-17T12:47:00.3067434Z "Expect `module_cls(*args, **kwargs)` returns an instance of , " 2022-08-17T12:47:00.3068859Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of , but it returns an instance of . 2022-08-17T12:47:00.3069586Z 2022-08-17T12:47:00.3069596Z 2022-08-17T12:47:00.6778411Z ok (2.170s) 2022-08-17T12:47:00.6778573Z 2022-08-17T12:47:00.6779037Z ---------------------------------------------------------------------- 2022-08-17T12:47:00.6779307Z Ran 1 test in 2.170s 2022-08-17T12:47:00.6779637Z 2022-08-17T12:47:00.6779705Z OK 2022-08-17T12:47:00.6779810Z 2022-08-17T12:47:00.6779909Z Generating XML reports... 2022-08-17T12:47:00.6815812Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124658.xml 2022-08-17T12:47:01.9993351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:01.9994035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:01.9997206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:01.9997795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:02.1539353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nh41pzs 2022-08-17T12:47:02.1543542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nh41pzs/_remote_module_non_scriptable.py 2022-08-17T12:47:02.4201002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:02.4212278Z 2022-08-17T12:47:02.4212622Z Running tests... 2022-08-17T12:47:02.4213320Z ---------------------------------------------------------------------- 2022-08-17T12:47:02.6889768Z test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31748 2022-08-17T12:47:02.6918853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31749 2022-08-17T12:47:03.6652975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:03.6653625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:03.6654464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:03.6655113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:03.6655854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:03.6656465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:03.6657486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:03.6658105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:03.8236675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwffv3k4 2022-08-17T12:47:03.8237818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwffv3k4/_remote_module_non_scriptable.py 2022-08-17T12:47:03.8249674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoghiywa1 2022-08-17T12:47:03.8252336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoghiywa1/_remote_module_non_scriptable.py 2022-08-17T12:47:04.0815736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:04.0866126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:04.2226449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwffv3k4/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:04.2230085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoghiywa1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:04.2297319Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpoghiywa1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:04.6967185Z ok (2.275s) 2022-08-17T12:47:04.6967407Z 2022-08-17T12:47:04.6968101Z ---------------------------------------------------------------------- 2022-08-17T12:47:04.6968359Z Ran 1 test in 2.275s 2022-08-17T12:47:04.6968481Z 2022-08-17T12:47:04.6968547Z OK 2022-08-17T12:47:04.6968646Z 2022-08-17T12:47:04.6968743Z Generating XML reports... 2022-08-17T12:47:04.7002279Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124702.xml 2022-08-17T12:47:06.0033885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:06.0034272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:06.0037001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:06.0037412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:06.1524822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe35ujp0q 2022-08-17T12:47:06.1529269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe35ujp0q/_remote_module_non_scriptable.py 2022-08-17T12:47:06.4165038Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:06.4176739Z 2022-08-17T12:47:06.4177110Z Running tests... 2022-08-17T12:47:06.4177546Z ---------------------------------------------------------------------- 2022-08-17T12:47:06.6853706Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31865 2022-08-17T12:47:06.6883106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31866 2022-08-17T12:47:07.6568161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:07.6568790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:07.6571332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:07.6571997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:07.6576959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:07.6577595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:07.6581696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:07.6582292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:07.8166961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxu2zh06 2022-08-17T12:47:07.8167701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqtsvq2f 2022-08-17T12:47:07.8168345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxu2zh06/_remote_module_non_scriptable.py 2022-08-17T12:47:07.8169001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqtsvq2f/_remote_module_non_scriptable.py 2022-08-17T12:47:08.0734446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:08.0749928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:08.1947955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqtsvq2f/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:08.1948591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxu2zh06/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:08.2017997Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpnqtsvq2f/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:08.6932533Z ok (2.275s) 2022-08-17T12:47:08.6932799Z 2022-08-17T12:47:08.6933332Z ---------------------------------------------------------------------- 2022-08-17T12:47:08.6933643Z Ran 1 test in 2.276s 2022-08-17T12:47:08.6933768Z 2022-08-17T12:47:08.6933839Z OK 2022-08-17T12:47:08.6933937Z 2022-08-17T12:47:08.6934036Z Generating XML reports... 2022-08-17T12:47:08.6967327Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124706.xml 2022-08-17T12:47:10.0079977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:10.0080408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:10.0082473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:10.0082857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:10.1606088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0jcnesi 2022-08-17T12:47:10.1610050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0jcnesi/_remote_module_non_scriptable.py 2022-08-17T12:47:10.4245375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:10.4256861Z 2022-08-17T12:47:10.4257227Z Running tests... 2022-08-17T12:47:10.4257665Z ---------------------------------------------------------------------- 2022-08-17T12:47:10.6901521Z test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31986 2022-08-17T12:47:10.6936852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31987 2022-08-17T12:47:11.6610544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:11.6611253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:11.6612068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:11.6612641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:11.6613623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:11.6614035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:11.6616539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:11.6617230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:11.8201131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qty0n5h 2022-08-17T12:47:11.8202201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qty0n5h/_remote_module_non_scriptable.py 2022-08-17T12:47:11.8202913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcy3rorbt 2022-08-17T12:47:11.8205303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy3rorbt/_remote_module_non_scriptable.py 2022-08-17T12:47:12.0773311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:12.0804772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:12.2104331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy3rorbt/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:12.2105456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qty0n5h/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:12.2152636Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp4qty0n5h/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:12.6984011Z ok (2.272s) 2022-08-17T12:47:12.6984249Z 2022-08-17T12:47:12.6984787Z ---------------------------------------------------------------------- 2022-08-17T12:47:12.6985116Z Ran 1 test in 2.273s 2022-08-17T12:47:12.6985239Z 2022-08-17T12:47:12.6985289Z OK 2022-08-17T12:47:12.6985385Z 2022-08-17T12:47:12.6985484Z Generating XML reports... 2022-08-17T12:47:12.7017807Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124710.xml 2022-08-17T12:47:14.0104017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:14.0104432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:14.0107034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:14.0107413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:14.1640111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4h109gp 2022-08-17T12:47:14.1643500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4h109gp/_remote_module_non_scriptable.py 2022-08-17T12:47:14.4304856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:14.4315739Z 2022-08-17T12:47:14.4315840Z Running tests... 2022-08-17T12:47:14.4316344Z ---------------------------------------------------------------------- 2022-08-17T12:47:14.6961451Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32107 2022-08-17T12:47:14.6990346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32108 2022-08-17T12:47:15.6569326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:15.6569706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:15.6573332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:15.6573990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:15.6693777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:15.6694159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:15.6697606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:15.6698244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:15.8129881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspvgkq5q 2022-08-17T12:47:15.8131559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspvgkq5q/_remote_module_non_scriptable.py 2022-08-17T12:47:15.8254540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5n52myx1 2022-08-17T12:47:15.8256374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5n52myx1/_remote_module_non_scriptable.py 2022-08-17T12:47:16.0712496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:16.0858651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:16.2021689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspvgkq5q/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:16.2026361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5n52myx1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:16.2094383Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp5n52myx1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:16.7037967Z ok (2.272s) 2022-08-17T12:47:16.7038151Z 2022-08-17T12:47:16.7038556Z ---------------------------------------------------------------------- 2022-08-17T12:47:16.7038963Z Ran 1 test in 2.272s 2022-08-17T12:47:16.7039073Z 2022-08-17T12:47:16.7039143Z OK 2022-08-17T12:47:16.7039241Z 2022-08-17T12:47:16.7039335Z Generating XML reports... 2022-08-17T12:47:16.7072660Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124714.xml 2022-08-17T12:47:18.0314713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:18.0315403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:18.0318166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:18.0318755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:18.1871590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a88uacx 2022-08-17T12:47:18.1874076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a88uacx/_remote_module_non_scriptable.py 2022-08-17T12:47:18.4548348Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:18.4559896Z 2022-08-17T12:47:18.4560382Z Running tests... 2022-08-17T12:47:18.4560803Z ---------------------------------------------------------------------- 2022-08-17T12:47:18.7200660Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32228 2022-08-17T12:47:18.7230666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32229 2022-08-17T12:47:19.6975744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:19.6976160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:19.6978401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:19.6978769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:19.7368988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:19.7369638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:19.7372999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:19.7373397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:19.8496183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yo6qhqz 2022-08-17T12:47:19.8498127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yo6qhqz/_remote_module_non_scriptable.py 2022-08-17T12:47:19.8921558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9za67fpo 2022-08-17T12:47:19.8923438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9za67fpo/_remote_module_non_scriptable.py 2022-08-17T12:47:20.1062981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:20.1509565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:20.6276586Z ok (2.171s) 2022-08-17T12:47:20.6276853Z 2022-08-17T12:47:20.6277358Z ---------------------------------------------------------------------- 2022-08-17T12:47:20.6277625Z Ran 1 test in 2.172s 2022-08-17T12:47:20.6277747Z 2022-08-17T12:47:20.6277799Z OK 2022-08-17T12:47:20.6277897Z 2022-08-17T12:47:20.6278244Z Generating XML reports... 2022-08-17T12:47:20.6310703Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124718.xml 2022-08-17T12:47:21.9311909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:21.9312598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:21.9316133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:21.9316736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:22.0860797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzjtrn0e 2022-08-17T12:47:22.0863817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzjtrn0e/_remote_module_non_scriptable.py 2022-08-17T12:47:22.3501797Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:22.3513129Z 2022-08-17T12:47:22.3513544Z Running tests... 2022-08-17T12:47:22.3514000Z ---------------------------------------------------------------------- 2022-08-17T12:47:22.6169093Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32345 2022-08-17T12:47:22.6198355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32346 2022-08-17T12:47:23.5908083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:23.5908789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:23.5911636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:23.5912194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:23.5913001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:23.5913613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:23.5916511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:23.5917180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:23.7462039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgqt9odm 2022-08-17T12:47:23.7463114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgqt9odm/_remote_module_non_scriptable.py 2022-08-17T12:47:23.7472385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_won0y75 2022-08-17T12:47:23.7475246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_won0y75/_remote_module_non_scriptable.py 2022-08-17T12:47:24.0052801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:24.0077099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:24.5242953Z ok (2.173s) 2022-08-17T12:47:24.5243219Z 2022-08-17T12:47:24.5243709Z ---------------------------------------------------------------------- 2022-08-17T12:47:24.5244219Z Ran 1 test in 2.173s 2022-08-17T12:47:24.5244409Z 2022-08-17T12:47:24.5244459Z OK 2022-08-17T12:47:24.5244556Z 2022-08-17T12:47:24.5244652Z Generating XML reports... 2022-08-17T12:47:24.5280518Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124722.xml 2022-08-17T12:47:25.8219139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:25.8219550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:25.8222659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:25.8223051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:25.9752803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng8i6hbo 2022-08-17T12:47:25.9756344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng8i6hbo/_remote_module_non_scriptable.py 2022-08-17T12:47:26.2442820Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:26.2453712Z 2022-08-17T12:47:26.2453848Z Running tests... 2022-08-17T12:47:26.2454504Z ---------------------------------------------------------------------- 2022-08-17T12:47:26.5107483Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32462 2022-08-17T12:47:26.5135583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32463 2022-08-17T12:47:27.4465761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:27.4466486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:27.4468707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:27.4469291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:27.4551198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:27.4551920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:27.4555640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:27.4556238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:27.6001518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq10wdmbm 2022-08-17T12:47:27.6002368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq10wdmbm/_remote_module_non_scriptable.py 2022-08-17T12:47:27.6085323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni7uxol8 2022-08-17T12:47:27.6086725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni7uxol8/_remote_module_non_scriptable.py 2022-08-17T12:47:27.8544228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:27.8630356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:28.3178798Z ok (2.072s) 2022-08-17T12:47:28.3178947Z 2022-08-17T12:47:28.3179298Z ---------------------------------------------------------------------- 2022-08-17T12:47:28.3179564Z Ran 1 test in 2.072s 2022-08-17T12:47:28.3179688Z 2022-08-17T12:47:28.3179757Z OK 2022-08-17T12:47:28.3179869Z 2022-08-17T12:47:28.3179951Z Generating XML reports... 2022-08-17T12:47:28.3216727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124726.xml 2022-08-17T12:47:29.6284597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:29.6284989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:29.6286840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:29.6287215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:29.7814542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov0hwsjj 2022-08-17T12:47:29.7818338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov0hwsjj/_remote_module_non_scriptable.py 2022-08-17T12:47:30.0460244Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:30.0471325Z 2022-08-17T12:47:30.0471720Z Running tests... 2022-08-17T12:47:30.0472152Z ---------------------------------------------------------------------- 2022-08-17T12:47:30.3151480Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32579 2022-08-17T12:47:30.3181520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32580 2022-08-17T12:47:31.2671987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:31.2672488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:31.2672947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:31.2673288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:31.2675321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:31.2675801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:31.2676274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:31.2676627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:31.4238056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdmy847z 2022-08-17T12:47:31.4239140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdmy847z/_remote_module_non_scriptable.py 2022-08-17T12:47:31.4245738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xsv15dp 2022-08-17T12:47:31.4248007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xsv15dp/_remote_module_non_scriptable.py 2022-08-17T12:47:31.6782161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:31.6782730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:31.8183592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xsv15dp/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:31.8184305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdmy847z/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:31.8254149Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp0xsv15dp/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T12:47:32.2227239Z ok (2.175s) 2022-08-17T12:47:32.2227493Z 2022-08-17T12:47:32.2228033Z ---------------------------------------------------------------------- 2022-08-17T12:47:32.2228312Z Ran 1 test in 2.176s 2022-08-17T12:47:32.2228436Z 2022-08-17T12:47:32.2228499Z OK 2022-08-17T12:47:32.2228594Z 2022-08-17T12:47:32.2228687Z Generating XML reports... 2022-08-17T12:47:32.2261786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124730.xml 2022-08-17T12:47:33.5286652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:33.5287047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:33.5289745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:33.5290251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:33.6803458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2zvqqrb 2022-08-17T12:47:33.6806827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2zvqqrb/_remote_module_non_scriptable.py 2022-08-17T12:47:33.9444777Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:33.9455727Z 2022-08-17T12:47:33.9455828Z Running tests... 2022-08-17T12:47:33.9456238Z ---------------------------------------------------------------------- 2022-08-17T12:47:34.2075273Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32696 2022-08-17T12:47:34.2104569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32697 2022-08-17T12:47:35.1856603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:35.1857250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:35.1859812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:35.1860453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:35.2073226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:35.2073900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:35.2077092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:35.2077702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:35.3425506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcptcbfg8 2022-08-17T12:47:35.3426859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcptcbfg8/_remote_module_non_scriptable.py 2022-08-17T12:47:35.3620792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5h_zwiq 2022-08-17T12:47:35.3621948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5h_zwiq/_remote_module_non_scriptable.py 2022-08-17T12:47:35.6063012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:35.6378224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:36.1150033Z ok (2.169s) 2022-08-17T12:47:36.1150296Z 2022-08-17T12:47:36.1150869Z ---------------------------------------------------------------------- 2022-08-17T12:47:36.1151245Z Ran 1 test in 2.169s 2022-08-17T12:47:36.1151367Z 2022-08-17T12:47:36.1151416Z OK 2022-08-17T12:47:36.1151510Z 2022-08-17T12:47:36.1151845Z Generating XML reports... 2022-08-17T12:47:36.1183989Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124733.xml 2022-08-17T12:47:37.4260755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:37.4261233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:37.4263782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:37.4264171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:37.5814912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fhlwjon 2022-08-17T12:47:37.5817747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fhlwjon/_remote_module_non_scriptable.py 2022-08-17T12:47:37.8530404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:37.8540989Z 2022-08-17T12:47:37.8541512Z Running tests... 2022-08-17T12:47:37.8541939Z ---------------------------------------------------------------------- 2022-08-17T12:47:38.1222438Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 345 2022-08-17T12:47:38.1250923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 346 2022-08-17T12:47:39.0605232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:39.0606110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:39.0607357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:39.0607973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:39.1160641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:39.1163245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:39.1163825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:39.1164202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:39.2146641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftxb3963 2022-08-17T12:47:39.2147467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftxb3963/_remote_module_non_scriptable.py 2022-08-17T12:47:39.2723834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13febar0 2022-08-17T12:47:39.2724819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13febar0/_remote_module_non_scriptable.py 2022-08-17T12:47:39.4752992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:39.5332993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:39.6855399Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``. 2022-08-17T12:47:40.0296578Z ok (2.175s) 2022-08-17T12:47:40.0296835Z 2022-08-17T12:47:40.0297377Z ---------------------------------------------------------------------- 2022-08-17T12:47:40.0297760Z Ran 1 test in 2.175s 2022-08-17T12:47:40.0297870Z 2022-08-17T12:47:40.0297935Z OK 2022-08-17T12:47:40.0298031Z 2022-08-17T12:47:40.0298124Z Generating XML reports... 2022-08-17T12:47:40.0331514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124737.xml 2022-08-17T12:47:41.3472686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:41.3473114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:41.3475893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:41.3476426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:41.4963414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm03mvnuo 2022-08-17T12:47:41.4967261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm03mvnuo/_remote_module_non_scriptable.py 2022-08-17T12:47:41.7588477Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:41.7598968Z 2022-08-17T12:47:41.7599120Z Running tests... 2022-08-17T12:47:41.7599753Z ---------------------------------------------------------------------- 2022-08-17T12:47:42.0182320Z test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 462 2022-08-17T12:47:42.0210200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 463 2022-08-17T12:47:42.9749658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:42.9750352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:42.9752966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:42.9753575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:42.9758965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:42.9759608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:42.9763484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:42.9764146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:43.1305471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ji0rwl8 2022-08-17T12:47:43.1306803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ji0rwl8/_remote_module_non_scriptable.py 2022-08-17T12:47:43.1318689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq641g7a2 2022-08-17T12:47:43.1321435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq641g7a2/_remote_module_non_scriptable.py 2022-08-17T12:47:43.3862243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:43.3894362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:43.9255719Z ok (2.165s) 2022-08-17T12:47:43.9255945Z 2022-08-17T12:47:43.9256502Z ---------------------------------------------------------------------- 2022-08-17T12:47:43.9256904Z Ran 1 test in 2.166s 2022-08-17T12:47:43.9257026Z 2022-08-17T12:47:43.9257090Z OK 2022-08-17T12:47:43.9257184Z 2022-08-17T12:47:43.9257266Z Generating XML reports... 2022-08-17T12:47:43.9290370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124741.xml 2022-08-17T12:47:45.2295185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:45.2295588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:45.2298049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:45.2298423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:45.3818857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppure163f 2022-08-17T12:47:45.3822040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppure163f/_remote_module_non_scriptable.py 2022-08-17T12:47:45.6462914Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:45.6473883Z 2022-08-17T12:47:45.6474031Z Running tests... 2022-08-17T12:47:45.6474643Z ---------------------------------------------------------------------- 2022-08-17T12:47:45.9254275Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 579 2022-08-17T12:47:45.9283718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 580 2022-08-17T12:47:46.9012641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:46.9013271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:46.9014720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:46.9015331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:46.9044109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:46.9044522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:46.9048007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:46.9048373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:47.0576540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzq1t_rj 2022-08-17T12:47:47.0577440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzq1t_rj/_remote_module_non_scriptable.py 2022-08-17T12:47:47.0598194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt05bp2ac 2022-08-17T12:47:47.0600642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt05bp2ac/_remote_module_non_scriptable.py 2022-08-17T12:47:47.3162070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:47.3162802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:47.8333034Z ok (2.186s) 2022-08-17T12:47:47.8333196Z 2022-08-17T12:47:47.8333639Z ---------------------------------------------------------------------- 2022-08-17T12:47:47.8333908Z Ran 1 test in 2.186s 2022-08-17T12:47:47.8334037Z 2022-08-17T12:47:47.8334088Z OK 2022-08-17T12:47:47.8334185Z 2022-08-17T12:47:47.8334279Z Generating XML reports... 2022-08-17T12:47:47.8370650Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124745.xml 2022-08-17T12:47:49.1337646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:49.1338043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:49.1340344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:49.1340707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:49.2832496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9k427ywg 2022-08-17T12:47:49.2835198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9k427ywg/_remote_module_non_scriptable.py 2022-08-17T12:47:49.5466367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:49.5476879Z 2022-08-17T12:47:49.5477017Z Running tests... 2022-08-17T12:47:49.5477428Z ---------------------------------------------------------------------- 2022-08-17T12:47:49.8097297Z test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 697 2022-08-17T12:47:49.8126362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 698 2022-08-17T12:47:49.8154579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 699 2022-08-17T12:47:49.8185770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 700 2022-08-17T12:47:50.8467237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:50.8467910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:50.8470822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:50.8471501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:50.8559144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:50.8559855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:50.8563835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:50.8564733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:50.9448176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:50.9448841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:50.9451656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:50.9452268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:50.9850205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:50.9850864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:50.9854032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:50.9854824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:51.0131188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveec927q 2022-08-17T12:47:51.0131985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveec927q/_remote_module_non_scriptable.py 2022-08-17T12:47:51.0190651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd6_ms4j 2022-08-17T12:47:51.0192225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd6_ms4j/_remote_module_non_scriptable.py 2022-08-17T12:47:51.1039975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf5x__sd 2022-08-17T12:47:51.1040888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf5x__sd/_remote_module_non_scriptable.py 2022-08-17T12:47:51.1454792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcntyf45f 2022-08-17T12:47:51.1455669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcntyf45f/_remote_module_non_scriptable.py 2022-08-17T12:47:51.2773814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:51.2848269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:51.3666022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:47:51.4102897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:47:52.0246841Z ok (2.477s) 2022-08-17T12:47:52.0247002Z 2022-08-17T12:47:52.0247326Z ---------------------------------------------------------------------- 2022-08-17T12:47:52.0247842Z Ran 1 test in 2.477s 2022-08-17T12:47:52.0247963Z 2022-08-17T12:47:52.0248030Z OK 2022-08-17T12:47:52.0248126Z 2022-08-17T12:47:52.0248222Z Generating XML reports... 2022-08-17T12:47:52.0280616Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124749.xml 2022-08-17T12:47:53.3131084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:53.3131488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:53.3133382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:53.3133744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:53.4604667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpbt26o9 2022-08-17T12:47:53.4607952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpbt26o9/_remote_module_non_scriptable.py 2022-08-17T12:47:53.7189784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:53.7200259Z 2022-08-17T12:47:53.7200374Z Running tests... 2022-08-17T12:47:53.7200949Z ---------------------------------------------------------------------- 2022-08-17T12:47:53.9785206Z test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 920 2022-08-17T12:47:53.9814205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 921 2022-08-17T12:47:53.9841487Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 922 2022-08-17T12:47:53.9874099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 923 2022-08-17T12:47:54.9944781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:54.9945543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:54.9948633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:54.9949302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:55.0496598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:55.0497228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:55.0500406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:55.0500948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:55.0997942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:55.0998629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:55.1001728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:55.1002423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:55.1191707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:55.1194970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:55.1195867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:55.1196528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:55.1567185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoaayul_ 2022-08-17T12:47:55.1568007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoaayul_/_remote_module_non_scriptable.py 2022-08-17T12:47:55.2108640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kcnf4jq 2022-08-17T12:47:55.2109603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kcnf4jq/_remote_module_non_scriptable.py 2022-08-17T12:47:55.2605439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6n2dyge 2022-08-17T12:47:55.2606696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6n2dyge/_remote_module_non_scriptable.py 2022-08-17T12:47:55.2784202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2twjo4_i 2022-08-17T12:47:55.2785210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2twjo4_i/_remote_module_non_scriptable.py 2022-08-17T12:47:55.4208334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:47:55.4715211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:55.5246631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:47:55.5410815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:56.1932908Z ok (2.473s) 2022-08-17T12:47:56.1933127Z 2022-08-17T12:47:56.1933767Z ---------------------------------------------------------------------- 2022-08-17T12:47:56.1934026Z Ran 1 test in 2.473s 2022-08-17T12:47:56.1934146Z 2022-08-17T12:47:56.1934216Z OK 2022-08-17T12:47:56.1934312Z 2022-08-17T12:47:56.1934407Z Generating XML reports... 2022-08-17T12:47:56.1967337Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124753.xml 2022-08-17T12:47:57.4764629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:57.4765019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:57.4767599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:57.4767996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:57.6285675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_dcfzwd 2022-08-17T12:47:57.6288964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_dcfzwd/_remote_module_non_scriptable.py 2022-08-17T12:47:57.8917154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:47:57.8927910Z 2022-08-17T12:47:57.8928133Z Running tests... 2022-08-17T12:47:57.8928614Z ---------------------------------------------------------------------- 2022-08-17T12:47:58.1531028Z test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1143 2022-08-17T12:47:58.1559772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1144 2022-08-17T12:47:58.1589441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1145 2022-08-17T12:47:58.1620168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1146 2022-08-17T12:47:59.1999893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:59.2000533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:59.2002893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:59.2003545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:59.2101823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:59.2102445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:59.2107179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:59.2108108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:59.2208957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:59.2209880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:59.2212903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:59.2213560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:59.2510766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:47:59.2511426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:47:59.2514365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:47:59.2515045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:47:59.3654344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbvb0kqy 2022-08-17T12:47:59.3655409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbvb0kqy/_remote_module_non_scriptable.py 2022-08-17T12:47:59.3836851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3sga84y 2022-08-17T12:47:59.3837686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3sga84y/_remote_module_non_scriptable.py 2022-08-17T12:47:59.3862876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwzslkqx 2022-08-17T12:47:59.3864391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwzslkqx/_remote_module_non_scriptable.py 2022-08-17T12:47:59.4106384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m_ojy6b 2022-08-17T12:47:59.4107141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m_ojy6b/_remote_module_non_scriptable.py 2022-08-17T12:47:59.6251833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:47:59.6433846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:47:59.6462231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:47:59.6709193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:00.2677824Z ok (2.375s) 2022-08-17T12:48:00.2678078Z 2022-08-17T12:48:00.2678624Z ---------------------------------------------------------------------- 2022-08-17T12:48:00.2678980Z Ran 1 test in 2.375s 2022-08-17T12:48:00.2688013Z 2022-08-17T12:48:00.2688141Z OK 2022-08-17T12:48:00.2688268Z 2022-08-17T12:48:00.2688369Z Generating XML reports... 2022-08-17T12:48:00.2711811Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124757.xml 2022-08-17T12:48:01.5667224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:01.5667624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:01.5670268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:01.5670671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:01.7215077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd8axxqd 2022-08-17T12:48:01.7218124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd8axxqd/_remote_module_non_scriptable.py 2022-08-17T12:48:01.9893425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:01.9904262Z 2022-08-17T12:48:01.9904640Z Running tests... 2022-08-17T12:48:01.9905285Z ---------------------------------------------------------------------- 2022-08-17T12:48:02.2562422Z test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1366 2022-08-17T12:48:02.2592451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1367 2022-08-17T12:48:02.2621246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1368 2022-08-17T12:48:02.2651665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1369 2022-08-17T12:48:03.2417254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:03.2417964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:03.2421133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:03.2713963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:03.2714977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:03.2715617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:03.2717166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:03.2717813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:03.3086359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:03.3086992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:03.3089438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:03.3090055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:03.3127173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:03.3127911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:03.3132420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:03.3133100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:03.3995379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7rhdi0t 2022-08-17T12:48:03.3996322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7rhdi0t/_remote_module_non_scriptable.py 2022-08-17T12:48:03.4321683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76cskr0z 2022-08-17T12:48:03.4322546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76cskr0z/_remote_module_non_scriptable.py 2022-08-17T12:48:03.4733571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp404mm1e 2022-08-17T12:48:03.4734886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp404mm1e/_remote_module_non_scriptable.py 2022-08-17T12:48:03.4876399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50xzzhu6 2022-08-17T12:48:03.4877694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50xzzhu6/_remote_module_non_scriptable.py 2022-08-17T12:48:03.6625113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:03.6974886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:03.7350279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:03.7501676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:04.3710737Z ok (2.380s) 2022-08-17T12:48:04.3711025Z 2022-08-17T12:48:04.3711900Z ---------------------------------------------------------------------- 2022-08-17T12:48:04.3712173Z Ran 1 test in 2.381s 2022-08-17T12:48:04.3712296Z 2022-08-17T12:48:04.3712359Z OK 2022-08-17T12:48:04.3712457Z 2022-08-17T12:48:04.3712538Z Generating XML reports... 2022-08-17T12:48:04.3748956Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124801.xml 2022-08-17T12:48:05.6583748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:05.6584128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:05.6586827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:05.6587201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:05.8072920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8hksjd_ 2022-08-17T12:48:05.8077772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8hksjd_/_remote_module_non_scriptable.py 2022-08-17T12:48:06.0674090Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:06.0684008Z 2022-08-17T12:48:06.0684468Z Running tests... 2022-08-17T12:48:06.0684837Z ---------------------------------------------------------------------- 2022-08-17T12:48:06.3257183Z test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1590 2022-08-17T12:48:06.3284997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1591 2022-08-17T12:48:06.3314397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1592 2022-08-17T12:48:06.3346707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1593 2022-08-17T12:48:07.3870750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:07.3871440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:07.3874799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:07.3875480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:07.3923933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:07.3924581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:07.3928224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:07.3928884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:07.3939309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:07.3939927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:07.3944487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:07.3945181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:07.4198685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:07.4199334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:07.4202255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:07.4203011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:07.5515569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mt0eoha 2022-08-17T12:48:07.5516407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mt0eoha/_remote_module_non_scriptable.py 2022-08-17T12:48:07.5523497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho8eqcwd 2022-08-17T12:48:07.5524947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho8eqcwd/_remote_module_non_scriptable.py 2022-08-17T12:48:07.5608444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiahx2oe3 2022-08-17T12:48:07.5610828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiahx2oe3/_remote_module_non_scriptable.py 2022-08-17T12:48:07.5795821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7guam5zq 2022-08-17T12:48:07.5796859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7guam5zq/_remote_module_non_scriptable.py 2022-08-17T12:48:07.8123693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:07.8373638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:07.8422569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:07.8443254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:15.2319233Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:48:17.1118130Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #5 to worker0, as the agent is shutting down 2022-08-17T12:48:17.4541016Z ok (11.385s) 2022-08-17T12:48:17.4541220Z 2022-08-17T12:48:17.4541584Z ---------------------------------------------------------------------- 2022-08-17T12:48:17.4541869Z Ran 1 test in 11.386s 2022-08-17T12:48:17.4541996Z 2022-08-17T12:48:17.4542093Z OK 2022-08-17T12:48:17.4542231Z 2022-08-17T12:48:17.4542346Z Generating XML reports... 2022-08-17T12:48:17.4576198Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124806.xml 2022-08-17T12:48:18.7525817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:18.7526492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:18.7529382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:18.7529857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:18.9048754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69tf7f3l 2022-08-17T12:48:18.9052177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69tf7f3l/_remote_module_non_scriptable.py 2022-08-17T12:48:19.1692853Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:19.1703735Z 2022-08-17T12:48:19.1703847Z Running tests... 2022-08-17T12:48:19.1704246Z ---------------------------------------------------------------------- 2022-08-17T12:48:19.4380971Z test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1813 2022-08-17T12:48:19.4408916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1814 2022-08-17T12:48:19.4437382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1815 2022-08-17T12:48:19.4468623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1816 2022-08-17T12:48:20.5080236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:20.5080852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:20.5083725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:20.5084684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:20.5178633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:20.5179303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:20.5182441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:20.5183129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:20.5474581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:20.5475222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:20.5477919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:20.5478517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:20.5486742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:20.5487416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:20.5491652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:20.5492398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:20.6671192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh6k2evu 2022-08-17T12:48:20.6673367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh6k2evu/_remote_module_non_scriptable.py 2022-08-17T12:48:20.6757031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz9_r78r 2022-08-17T12:48:20.6759594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz9_r78r/_remote_module_non_scriptable.py 2022-08-17T12:48:20.7073784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qxue8ry 2022-08-17T12:48:20.7074820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qxue8ry/_remote_module_non_scriptable.py 2022-08-17T12:48:20.7161427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1jiqn1k 2022-08-17T12:48:20.7162912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1jiqn1k/_remote_module_non_scriptable.py 2022-08-17T12:48:20.9268367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:20.9412234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:20.9670087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:20.9765627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:21.6529387Z ok (2.482s) 2022-08-17T12:48:21.6529609Z 2022-08-17T12:48:21.6530081Z ---------------------------------------------------------------------- 2022-08-17T12:48:21.6530513Z Ran 1 test in 2.482s 2022-08-17T12:48:21.6530691Z 2022-08-17T12:48:21.6530798Z OK 2022-08-17T12:48:21.6530954Z 2022-08-17T12:48:21.6531109Z Generating XML reports... 2022-08-17T12:48:21.6564931Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124819.xml 2022-08-17T12:48:22.9661852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:22.9662242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:22.9665103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:22.9665581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:23.1225946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ip0726_ 2022-08-17T12:48:23.1229297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ip0726_/_remote_module_non_scriptable.py 2022-08-17T12:48:23.3900388Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:23.3911251Z 2022-08-17T12:48:23.3911564Z Running tests... 2022-08-17T12:48:23.3912231Z ---------------------------------------------------------------------- 2022-08-17T12:48:23.6547570Z test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2036 2022-08-17T12:48:23.6575552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2037 2022-08-17T12:48:23.6603584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2038 2022-08-17T12:48:23.6636907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2039 2022-08-17T12:48:24.7182019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:24.7182755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:24.7186049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:24.7186751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:24.7266994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:24.7267619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:24.7272122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:24.7272745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:24.7699444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:24.7700191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:24.7703055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:24.7703744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:24.7771007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:24.7771565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:24.7775043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:24.7775600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:24.8783741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpku04wg4t 2022-08-17T12:48:24.8784750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpku04wg4t/_remote_module_non_scriptable.py 2022-08-17T12:48:24.8893717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowfadr7m 2022-08-17T12:48:24.8895936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowfadr7m/_remote_module_non_scriptable.py 2022-08-17T12:48:24.9301644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41laryja 2022-08-17T12:48:24.9302711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41laryja/_remote_module_non_scriptable.py 2022-08-17T12:48:24.9366697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r6vvpzc 2022-08-17T12:48:24.9369048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r6vvpzc/_remote_module_non_scriptable.py 2022-08-17T12:48:25.1509050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:25.1547715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:25.1920704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:25.1976556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:25.8697442Z ok (2.478s) 2022-08-17T12:48:25.8697606Z 2022-08-17T12:48:25.8698017Z ---------------------------------------------------------------------- 2022-08-17T12:48:25.8698303Z Ran 1 test in 2.479s 2022-08-17T12:48:25.8698430Z 2022-08-17T12:48:25.8698495Z OK 2022-08-17T12:48:25.8698591Z 2022-08-17T12:48:25.8698674Z Generating XML reports... 2022-08-17T12:48:25.8732680Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124823.xml 2022-08-17T12:48:27.1869854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:27.1870247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:27.1872572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:27.1873178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:27.3393512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bh1u16z 2022-08-17T12:48:27.3396190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bh1u16z/_remote_module_non_scriptable.py 2022-08-17T12:48:27.6067338Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:27.6078533Z 2022-08-17T12:48:27.6078842Z Running tests... 2022-08-17T12:48:27.6079511Z ---------------------------------------------------------------------- 2022-08-17T12:48:27.8753341Z test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2259 2022-08-17T12:48:27.8780831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2260 2022-08-17T12:48:27.8809063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2261 2022-08-17T12:48:27.8840533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2262 2022-08-17T12:48:28.9244606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:28.9245245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:28.9248509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:28.9249152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:28.9349937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:28.9350604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:28.9354324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:28.9355017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:28.9631969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:28.9632582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:28.9635987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:28.9636642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:28.9745868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:28.9746859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:28.9749905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:28.9750497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:29.0830380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7n029r1 2022-08-17T12:48:29.0831437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7n029r1/_remote_module_non_scriptable.py 2022-08-17T12:48:29.0955796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nidrvja 2022-08-17T12:48:29.0956574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nidrvja/_remote_module_non_scriptable.py 2022-08-17T12:48:29.1221683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53yvvg_n 2022-08-17T12:48:29.1222506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53yvvg_n/_remote_module_non_scriptable.py 2022-08-17T12:48:29.1317934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tnc11cm 2022-08-17T12:48:29.1318854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tnc11cm/_remote_module_non_scriptable.py 2022-08-17T12:48:29.3452112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:29.3613407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:29.3857997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:29.3917398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:30.0901167Z ok (2.482s) 2022-08-17T12:48:30.0901448Z 2022-08-17T12:48:30.0901912Z ---------------------------------------------------------------------- 2022-08-17T12:48:30.0902165Z Ran 1 test in 2.482s 2022-08-17T12:48:30.0902304Z 2022-08-17T12:48:30.0902868Z OK 2022-08-17T12:48:30.0903033Z 2022-08-17T12:48:30.0903145Z Generating XML reports... 2022-08-17T12:48:30.0937321Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124827.xml 2022-08-17T12:48:31.3949747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:31.3950151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:31.3952510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:31.3952873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:31.5487437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhx791yx 2022-08-17T12:48:31.5490828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhx791yx/_remote_module_non_scriptable.py 2022-08-17T12:48:31.8155651Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:31.8166536Z 2022-08-17T12:48:31.8166622Z Running tests... 2022-08-17T12:48:31.8167086Z ---------------------------------------------------------------------- 2022-08-17T12:48:32.0886106Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2482 2022-08-17T12:48:32.0917163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2483 2022-08-17T12:48:32.0945755Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2484 2022-08-17T12:48:32.0977434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2485 2022-08-17T12:48:33.1798664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:33.1799302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:33.1801587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:33.1802166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:33.2005106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:33.2005734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:33.2008982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:33.2009618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:33.2071380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:33.2072050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:33.2075117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:33.2075768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:33.2285235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:33.2286086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:33.2288950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:33.2289603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:33.3395892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno1gqfez 2022-08-17T12:48:33.3396754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno1gqfez/_remote_module_non_scriptable.py 2022-08-17T12:48:33.3612914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxfbfjhd 2022-08-17T12:48:33.3614237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxfbfjhd/_remote_module_non_scriptable.py 2022-08-17T12:48:33.3646735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eh7s5qz 2022-08-17T12:48:33.3648426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eh7s5qz/_remote_module_non_scriptable.py 2022-08-17T12:48:33.3856624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9w5hxzz7 2022-08-17T12:48:33.3857694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9w5hxzz7/_remote_module_non_scriptable.py 2022-08-17T12:48:33.6004826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:33.6228128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:33.6244722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:33.6474035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:34.3037641Z ok (2.487s) 2022-08-17T12:48:34.3037867Z 2022-08-17T12:48:34.3038430Z ---------------------------------------------------------------------- 2022-08-17T12:48:34.3038839Z Ran 1 test in 2.487s 2022-08-17T12:48:34.3038961Z 2022-08-17T12:48:34.3039026Z OK 2022-08-17T12:48:34.3039108Z 2022-08-17T12:48:34.3039204Z Generating XML reports... 2022-08-17T12:48:34.3072758Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124831.xml 2022-08-17T12:48:35.6286661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:35.6287052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:35.6290011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:35.7803473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:35.7803867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bgga1hh 2022-08-17T12:48:35.7807441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bgga1hh/_remote_module_non_scriptable.py 2022-08-17T12:48:36.0475620Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:36.0485535Z 2022-08-17T12:48:36.0485798Z Running tests... 2022-08-17T12:48:36.0486395Z ---------------------------------------------------------------------- 2022-08-17T12:48:36.3250905Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2705 2022-08-17T12:48:36.3278661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2706 2022-08-17T12:48:36.3307068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2707 2022-08-17T12:48:36.3337917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2708 2022-08-17T12:48:37.3456063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:37.3456997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:37.3458565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:37.3459219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:37.3554272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:37.3554948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:37.3559248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:37.3559943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:37.3650961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:37.3651616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:37.3654905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:37.3655584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:37.4073255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:37.4073957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:37.4076952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:37.4077666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:37.5057609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphze70r1o 2022-08-17T12:48:37.5058627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphze70r1o/_remote_module_non_scriptable.py 2022-08-17T12:48:37.5162366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7fh_apa 2022-08-17T12:48:37.5163572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7fh_apa/_remote_module_non_scriptable.py 2022-08-17T12:48:37.5240819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3kv0upr 2022-08-17T12:48:37.5241985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3kv0upr/_remote_module_non_scriptable.py 2022-08-17T12:48:37.5660490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xx9vcoj 2022-08-17T12:48:37.5661661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xx9vcoj/_remote_module_non_scriptable.py 2022-08-17T12:48:37.7668740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:37.7808164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:37.7912417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:37.8294878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:38.5398396Z ok (2.491s) 2022-08-17T12:48:38.5398665Z 2022-08-17T12:48:38.5399092Z ---------------------------------------------------------------------- 2022-08-17T12:48:38.5399359Z Ran 1 test in 2.491s 2022-08-17T12:48:38.5399480Z 2022-08-17T12:48:38.5399545Z OK 2022-08-17T12:48:38.5399641Z 2022-08-17T12:48:38.5399736Z Generating XML reports... 2022-08-17T12:48:38.5435120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124836.xml 2022-08-17T12:48:39.8123407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:39.8123807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:39.8126835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:39.8127231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:39.9627438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zb2xed2 2022-08-17T12:48:39.9630668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zb2xed2/_remote_module_non_scriptable.py 2022-08-17T12:48:40.2241599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:40.2252580Z 2022-08-17T12:48:40.2252678Z Running tests... 2022-08-17T12:48:40.2253161Z ---------------------------------------------------------------------- 2022-08-17T12:48:40.4979400Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2928 2022-08-17T12:48:40.5007638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2929 2022-08-17T12:48:40.5037099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2930 2022-08-17T12:48:40.5070542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2931 2022-08-17T12:48:41.5551712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:41.5552378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:41.5553777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:41.5554420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:41.5734870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:41.5735523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:41.5738854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:41.5739536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:41.5764833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:41.5765452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:41.5769502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:41.5770157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:41.5833021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:41.5833648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:41.5837544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:41.5838236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:41.7121742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuridj4a 2022-08-17T12:48:41.7122557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuridj4a/_remote_module_non_scriptable.py 2022-08-17T12:48:41.7341763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9l4_jdt 2022-08-17T12:48:41.7343036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9l4_jdt/_remote_module_non_scriptable.py 2022-08-17T12:48:41.7349185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogd4sb7d 2022-08-17T12:48:41.7351685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogd4sb7d/_remote_module_non_scriptable.py 2022-08-17T12:48:41.7411528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81ojhe1g 2022-08-17T12:48:41.7413697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81ojhe1g/_remote_module_non_scriptable.py 2022-08-17T12:48:41.9712528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:41.9938201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:41.9943919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:41.9989797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:42.7129795Z ok (2.487s) 2022-08-17T12:48:42.7130072Z 2022-08-17T12:48:42.7130585Z ---------------------------------------------------------------------- 2022-08-17T12:48:42.7130869Z Ran 1 test in 2.488s 2022-08-17T12:48:42.7130990Z 2022-08-17T12:48:42.7131055Z OK 2022-08-17T12:48:42.7131155Z 2022-08-17T12:48:42.7131250Z Generating XML reports... 2022-08-17T12:48:42.7167414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124840.xml 2022-08-17T12:48:44.0116841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:44.0117538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:44.0120803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:44.0121401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:44.1639787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld7hs6r7 2022-08-17T12:48:44.1643024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld7hs6r7/_remote_module_non_scriptable.py 2022-08-17T12:48:44.4304928Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:44.4315653Z 2022-08-17T12:48:44.4315963Z Running tests... 2022-08-17T12:48:44.4316652Z ---------------------------------------------------------------------- 2022-08-17T12:48:44.6936730Z test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3151 2022-08-17T12:48:44.6966897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3152 2022-08-17T12:48:44.6997462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3153 2022-08-17T12:48:44.7032042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3154 2022-08-17T12:48:45.6810150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:45.6811160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:45.6813913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:45.6814590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:45.7332444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:45.7333121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:45.7335883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:45.7336550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:45.7548866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:45.7549541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:45.7552071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:45.7552705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:45.7616891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:45.7617573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:45.7621881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:45.7622540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:45.8404062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvvhigsh 2022-08-17T12:48:45.8405554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvvhigsh/_remote_module_non_scriptable.py 2022-08-17T12:48:45.8937024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3w5i0a4r 2022-08-17T12:48:45.8937843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3w5i0a4r/_remote_module_non_scriptable.py 2022-08-17T12:48:45.9151267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgaoft2m 2022-08-17T12:48:45.9152768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgaoft2m/_remote_module_non_scriptable.py 2022-08-17T12:48:45.9214570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbs4npwv 2022-08-17T12:48:45.9215669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbs4npwv/_remote_module_non_scriptable.py 2022-08-17T12:48:46.1026956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:46.1533956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:46.1797843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:46.1852776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:46.8093348Z ok (2.377s) 2022-08-17T12:48:46.8093643Z 2022-08-17T12:48:46.8094163Z ---------------------------------------------------------------------- 2022-08-17T12:48:46.8094573Z Ran 1 test in 2.378s 2022-08-17T12:48:46.8094759Z 2022-08-17T12:48:46.8094858Z OK 2022-08-17T12:48:46.8094993Z 2022-08-17T12:48:46.8095150Z Generating XML reports... 2022-08-17T12:48:46.8128685Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124844.xml 2022-08-17T12:48:48.1220926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:48.1221316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:48.1223534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:48.1224049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:48.2751608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jmscuej 2022-08-17T12:48:48.2754816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jmscuej/_remote_module_non_scriptable.py 2022-08-17T12:48:48.5391426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:48.5401588Z 2022-08-17T12:48:48.5401723Z Running tests... 2022-08-17T12:48:48.5402529Z ---------------------------------------------------------------------- 2022-08-17T12:48:48.8047962Z test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3374 2022-08-17T12:48:48.8076900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3375 2022-08-17T12:48:48.8106236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3376 2022-08-17T12:48:48.8140783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3377 2022-08-17T12:48:49.7876466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:49.7877124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:49.7878801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:49.7879411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:49.8020819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:49.8021495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:49.8024691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:49.8025423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:49.8225191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:49.8225868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:49.8228973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:49.8229628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:49.8621488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:49.8622088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:49.8625096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:49.8625653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:49.9454787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkrk8hqc 2022-08-17T12:48:49.9455641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkrk8hqc/_remote_module_non_scriptable.py 2022-08-17T12:48:49.9597671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe8wnm0y 2022-08-17T12:48:49.9598444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe8wnm0y/_remote_module_non_scriptable.py 2022-08-17T12:48:49.9811245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg24ica7m 2022-08-17T12:48:49.9812071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg24ica7m/_remote_module_non_scriptable.py 2022-08-17T12:48:50.0191576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_a8y14o 2022-08-17T12:48:50.0192520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_a8y14o/_remote_module_non_scriptable.py 2022-08-17T12:48:50.2068435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:50.2196859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:50.2434598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:50.2827471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:50.9198558Z ok (2.379s) 2022-08-17T12:48:50.9198848Z 2022-08-17T12:48:50.9199368Z ---------------------------------------------------------------------- 2022-08-17T12:48:50.9199652Z Ran 1 test in 2.380s 2022-08-17T12:48:50.9199775Z 2022-08-17T12:48:50.9199841Z OK 2022-08-17T12:48:50.9199938Z 2022-08-17T12:48:50.9200035Z Generating XML reports... 2022-08-17T12:48:50.9235278Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124848.xml 2022-08-17T12:48:52.2177227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:52.2177850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:52.2180059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:52.2180454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:52.3688270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5n7v1ucz 2022-08-17T12:48:52.3690750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5n7v1ucz/_remote_module_non_scriptable.py 2022-08-17T12:48:52.6327174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:52.6337140Z 2022-08-17T12:48:52.6337252Z Running tests... 2022-08-17T12:48:52.6338207Z ---------------------------------------------------------------------- 2022-08-17T12:48:52.8987466Z test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3597 2022-08-17T12:48:52.9014714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3598 2022-08-17T12:48:52.9045179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3599 2022-08-17T12:48:52.9074079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3600 2022-08-17T12:48:53.9015479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:53.9016087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:53.9019192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:53.9019872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:53.9074952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:53.9075610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:53.9079223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:53.9079883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:53.9217856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:53.9218517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:53.9222016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:53.9222949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:53.9567363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:53.9568011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:53.9570517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:53.9571169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:54.0638434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf81296mx 2022-08-17T12:48:54.0639672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf81296mx/_remote_module_non_scriptable.py 2022-08-17T12:48:54.0714661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmozdi63 2022-08-17T12:48:54.0716298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmozdi63/_remote_module_non_scriptable.py 2022-08-17T12:48:54.0831136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppl84lyuj 2022-08-17T12:48:54.0832265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppl84lyuj/_remote_module_non_scriptable.py 2022-08-17T12:48:54.1150907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbtqf6mq 2022-08-17T12:48:54.1152220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbtqf6mq/_remote_module_non_scriptable.py 2022-08-17T12:48:54.3237460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:54.3300970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:54.3456728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:54.3773828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:55.0132342Z ok (2.379s) 2022-08-17T12:48:55.0132597Z 2022-08-17T12:48:55.0133114Z ---------------------------------------------------------------------- 2022-08-17T12:48:55.0133601Z Ran 1 test in 2.379s 2022-08-17T12:48:55.0133821Z 2022-08-17T12:48:55.0133934Z OK 2022-08-17T12:48:55.0134106Z 2022-08-17T12:48:55.0134292Z Generating XML reports... 2022-08-17T12:48:55.0166634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124852.xml 2022-08-17T12:48:56.3014996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:56.3015381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:56.3018092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:56.3018471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:56.4540337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprft_qlm8 2022-08-17T12:48:56.4543798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprft_qlm8/_remote_module_non_scriptable.py 2022-08-17T12:48:56.7224047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:48:56.7235341Z 2022-08-17T12:48:56.7235698Z Running tests... 2022-08-17T12:48:56.7236069Z ---------------------------------------------------------------------- 2022-08-17T12:48:56.9929125Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3820 2022-08-17T12:48:56.9960257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3821 2022-08-17T12:48:56.9990856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3822 2022-08-17T12:48:57.0031794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3823 2022-08-17T12:48:58.0343874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:58.0344907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:58.0347334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:58.0348156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:58.0442628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:58.0443292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:58.0446966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:58.0447602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:58.0838958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:58.0839488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:58.0843000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:58.0843557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:58.0887543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:48:58.0888228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:48:58.0893200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:48:58.0893844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:48:58.1948836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xljj9wi 2022-08-17T12:48:58.1949675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xljj9wi/_remote_module_non_scriptable.py 2022-08-17T12:48:58.2011084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmhvf1_p 2022-08-17T12:48:58.2012272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmhvf1_p/_remote_module_non_scriptable.py 2022-08-17T12:48:58.2416996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt0m2q8c 2022-08-17T12:48:58.2417754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt0m2q8c/_remote_module_non_scriptable.py 2022-08-17T12:48:58.2489852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuvhgs8r 2022-08-17T12:48:58.2491416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuvhgs8r/_remote_module_non_scriptable.py 2022-08-17T12:48:58.4569338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:48:58.4587156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:48:58.5017901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:48:58.5082323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:48:59.1089423Z ok (2.385s) 2022-08-17T12:48:59.1089675Z 2022-08-17T12:48:59.1090011Z ---------------------------------------------------------------------- 2022-08-17T12:48:59.1090321Z Ran 1 test in 2.385s 2022-08-17T12:48:59.1090467Z 2022-08-17T12:48:59.1090533Z OK 2022-08-17T12:48:59.1090629Z 2022-08-17T12:48:59.1090713Z Generating XML reports... 2022-08-17T12:48:59.1123598Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124856.xml 2022-08-17T12:49:00.4102179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:00.4102846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:00.4105153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:00.4105544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:00.5609495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqxut3c7 2022-08-17T12:49:00.5613306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqxut3c7/_remote_module_non_scriptable.py 2022-08-17T12:49:00.8246470Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:00.8258057Z 2022-08-17T12:49:00.8258505Z Running tests... 2022-08-17T12:49:00.8258896Z ---------------------------------------------------------------------- 2022-08-17T12:49:01.0901230Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4043 2022-08-17T12:49:01.0928747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4044 2022-08-17T12:49:01.0957025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4045 2022-08-17T12:49:01.0990986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4046 2022-08-17T12:49:02.1781802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:02.1782471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:02.1786884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:02.1787535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:02.1857010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:02.1857667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:02.1861962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:02.1863014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:02.2015212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:02.2015852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:02.2019494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:02.2020183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:02.2522013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:02.2522587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:02.2525563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:02.2526321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:02.3533702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepgh9asi 2022-08-17T12:49:02.3534417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepgh9asi/_remote_module_non_scriptable.py 2022-08-17T12:49:02.3541906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbolrhrok 2022-08-17T12:49:02.3543638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbolrhrok/_remote_module_non_scriptable.py 2022-08-17T12:49:02.3635327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyosdfl2 2022-08-17T12:49:02.3636742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyosdfl2/_remote_module_non_scriptable.py 2022-08-17T12:49:02.4109478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2bgpe18 2022-08-17T12:49:02.4110645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2bgpe18/_remote_module_non_scriptable.py 2022-08-17T12:49:02.6172551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:02.6183952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:02.6281474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:02.6759579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:03.3049273Z ok (2.479s) 2022-08-17T12:49:03.3049544Z 2022-08-17T12:49:03.3050082Z ---------------------------------------------------------------------- 2022-08-17T12:49:03.3050367Z Ran 1 test in 2.479s 2022-08-17T12:49:03.3050513Z 2022-08-17T12:49:03.3050578Z OK 2022-08-17T12:49:03.3050675Z 2022-08-17T12:49:03.3050775Z Generating XML reports... 2022-08-17T12:49:03.3082939Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124900.xml 2022-08-17T12:49:04.5938117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:04.5938529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:04.5940723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:04.5941295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:04.7445683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhddq3u9 2022-08-17T12:49:04.7449108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhddq3u9/_remote_module_non_scriptable.py 2022-08-17T12:49:05.0095988Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:05.0106406Z 2022-08-17T12:49:05.0106536Z Running tests... 2022-08-17T12:49:05.0107148Z ---------------------------------------------------------------------- 2022-08-17T12:49:05.2706631Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4266 2022-08-17T12:49:05.2736044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4267 2022-08-17T12:49:05.2764382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4268 2022-08-17T12:49:05.2796275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4269 2022-08-17T12:49:06.3001754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:06.3002411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:06.3005057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:06.3005715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:06.3075322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:06.3075918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:06.3079518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:06.3080113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:06.3185775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:06.3186370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:06.3189169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:06.3189829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:06.3226997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:06.3227578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:06.3232177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:06.3232818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:06.4611098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsn83a5j 2022-08-17T12:49:06.4611957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsn83a5j/_remote_module_non_scriptable.py 2022-08-17T12:49:06.4775856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2zj1180 2022-08-17T12:49:06.4777270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2zj1180/_remote_module_non_scriptable.py 2022-08-17T12:49:06.4815484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xelz2x7 2022-08-17T12:49:06.4816679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xelz2x7/_remote_module_non_scriptable.py 2022-08-17T12:49:06.4857943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokesd7ms 2022-08-17T12:49:06.4859462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokesd7ms/_remote_module_non_scriptable.py 2022-08-17T12:49:06.7242847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:06.7393421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:06.7423712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:06.7441887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:07.3854909Z ok (2.375s) 2022-08-17T12:49:07.3855194Z 2022-08-17T12:49:07.3855732Z ---------------------------------------------------------------------- 2022-08-17T12:49:07.3856208Z Ran 1 test in 2.375s 2022-08-17T12:49:07.3856334Z 2022-08-17T12:49:07.3856399Z OK 2022-08-17T12:49:07.3856480Z 2022-08-17T12:49:07.3856580Z Generating XML reports... 2022-08-17T12:49:07.3889020Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124905.xml 2022-08-17T12:49:08.6751956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:08.6752372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:08.6754680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:08.6755071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:08.8268845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i7yeo1f 2022-08-17T12:49:08.8271850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i7yeo1f/_remote_module_non_scriptable.py 2022-08-17T12:49:09.0904382Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:09.0914962Z 2022-08-17T12:49:09.0915047Z Running tests... 2022-08-17T12:49:09.0915534Z ---------------------------------------------------------------------- 2022-08-17T12:49:09.3524704Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4489 2022-08-17T12:49:09.3555623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4490 2022-08-17T12:49:09.3584474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4491 2022-08-17T12:49:09.3615230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4492 2022-08-17T12:49:10.4515424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:10.4518608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:10.4519445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:10.4520078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:10.4705028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:10.4705677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:10.4708803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:10.4709486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:10.4833894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:10.4834889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:10.4838401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:10.4839036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:10.4957972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:10.4958686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:10.4962303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:10.4963023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:10.6191957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cpuoccw 2022-08-17T12:49:10.6192834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cpuoccw/_remote_module_non_scriptable.py 2022-08-17T12:49:10.6320521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr58wgm3t 2022-08-17T12:49:10.6321357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr58wgm3t/_remote_module_non_scriptable.py 2022-08-17T12:49:10.6557263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrma6ukr 2022-08-17T12:49:10.6558282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrma6ukr/_remote_module_non_scriptable.py 2022-08-17T12:49:10.6606754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypy72ud4 2022-08-17T12:49:10.6608296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypy72ud4/_remote_module_non_scriptable.py 2022-08-17T12:49:10.8806327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:10.8958094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:10.9182379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:10.9256232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:11.5675653Z ok (2.476s) 2022-08-17T12:49:11.5675917Z 2022-08-17T12:49:11.5676425Z ---------------------------------------------------------------------- 2022-08-17T12:49:11.5676695Z Ran 1 test in 2.476s 2022-08-17T12:49:11.5676818Z 2022-08-17T12:49:11.5676883Z OK 2022-08-17T12:49:11.5676980Z 2022-08-17T12:49:11.5677073Z Generating XML reports... 2022-08-17T12:49:11.5709324Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124909.xml 2022-08-17T12:49:12.8804393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:12.8805049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:12.8807619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:12.8808214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:13.0304564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv9619xm 2022-08-17T12:49:13.0308499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv9619xm/_remote_module_non_scriptable.py 2022-08-17T12:49:13.2911939Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:13.2923241Z 2022-08-17T12:49:13.2923525Z Running tests... 2022-08-17T12:49:13.2924214Z ---------------------------------------------------------------------- 2022-08-17T12:49:13.5515910Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4712 2022-08-17T12:49:13.5545977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4713 2022-08-17T12:49:13.5574318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4714 2022-08-17T12:49:13.5607525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4715 2022-08-17T12:49:14.5693300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:14.5693956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:14.5695547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:14.5696131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:14.6045471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:14.6046347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:14.6049110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:14.6049786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:14.6190877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:14.6191446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:14.6195126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:14.6195656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:14.6207340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:14.6207850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:14.6213467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:14.6214011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:14.7325808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4u_f3w4 2022-08-17T12:49:14.7326815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4u_f3w4/_remote_module_non_scriptable.py 2022-08-17T12:49:14.7661241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ycbdje2 2022-08-17T12:49:14.7662310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ycbdje2/_remote_module_non_scriptable.py 2022-08-17T12:49:14.7800831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz46l94l 2022-08-17T12:49:14.7802368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz46l94l/_remote_module_non_scriptable.py 2022-08-17T12:49:14.7959944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmv8ic8r 2022-08-17T12:49:14.7960977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmv8ic8r/_remote_module_non_scriptable.py 2022-08-17T12:49:14.9931674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:15.0277144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:15.0437035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:15.0587005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:15.6665091Z ok (2.374s) 2022-08-17T12:49:15.6665400Z 2022-08-17T12:49:15.6665803Z ---------------------------------------------------------------------- 2022-08-17T12:49:15.6666071Z Ran 1 test in 2.374s 2022-08-17T12:49:15.6666193Z 2022-08-17T12:49:15.6666243Z OK 2022-08-17T12:49:15.6666341Z 2022-08-17T12:49:15.6666437Z Generating XML reports... 2022-08-17T12:49:15.6701426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124913.xml 2022-08-17T12:49:16.9727540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:16.9727950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:16.9729903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:16.9730419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:17.1277768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2lwk6cn 2022-08-17T12:49:17.1280788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2lwk6cn/_remote_module_non_scriptable.py 2022-08-17T12:49:17.3929892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:17.3940929Z 2022-08-17T12:49:17.3941123Z Running tests... 2022-08-17T12:49:17.3941534Z ---------------------------------------------------------------------- 2022-08-17T12:49:17.6578060Z test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4935 2022-08-17T12:49:17.6606775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4936 2022-08-17T12:49:17.6635344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4937 2022-08-17T12:49:17.6666273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4938 2022-08-17T12:49:18.7167013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:18.7167698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:18.7170120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:18.7170758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:18.7492522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:18.7493117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:18.7495914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:18.7496516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:18.8463429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:18.8464438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:18.8466172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:18.8466860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:18.8478715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:18.8479383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:18.8484174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:18.8484784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:18.8780094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbil4t6uv 2022-08-17T12:49:18.8781334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbil4t6uv/_remote_module_non_scriptable.py 2022-08-17T12:49:18.9284299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5o6w4z5 2022-08-17T12:49:18.9285381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5o6w4z5/_remote_module_non_scriptable.py 2022-08-17T12:49:19.0031611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5tjr50r 2022-08-17T12:49:19.0032410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5tjr50r/_remote_module_non_scriptable.py 2022-08-17T12:49:19.0066261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gcj9g3s 2022-08-17T12:49:19.0067076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gcj9g3s/_remote_module_non_scriptable.py 2022-08-17T12:49:19.1363799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:19.1895296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:19.2632598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:19.2673048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:19.9730116Z ok (2.578s) 2022-08-17T12:49:19.9730289Z 2022-08-17T12:49:19.9730616Z ---------------------------------------------------------------------- 2022-08-17T12:49:19.9730893Z Ran 1 test in 2.579s 2022-08-17T12:49:19.9731015Z 2022-08-17T12:49:19.9731065Z OK 2022-08-17T12:49:19.9731167Z 2022-08-17T12:49:19.9731261Z Generating XML reports... 2022-08-17T12:49:19.9763646Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124917.xml 2022-08-17T12:49:21.2466762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:21.2467479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:21.2470538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:21.2471130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:21.3993163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm9gitzi 2022-08-17T12:49:21.3996375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm9gitzi/_remote_module_non_scriptable.py 2022-08-17T12:49:21.6648960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:21.6660292Z 2022-08-17T12:49:21.6660747Z Running tests... 2022-08-17T12:49:21.6661172Z ---------------------------------------------------------------------- 2022-08-17T12:49:21.9301254Z test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5158 2022-08-17T12:49:21.9331647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5159 2022-08-17T12:49:21.9362553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5160 2022-08-17T12:49:21.9395501Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5161 2022-08-17T12:49:22.9222376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:22.9223056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:22.9224621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:22.9225343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:22.9674486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:22.9675174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:22.9678398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:22.9679004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:22.9883697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:22.9884390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:22.9887460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:22.9888133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:23.0593587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:23.0594234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:23.0597078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:23.0597753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:23.0824329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaaxgqg2 2022-08-17T12:49:23.0825257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaaxgqg2/_remote_module_non_scriptable.py 2022-08-17T12:49:23.1260367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1dp_wb5 2022-08-17T12:49:23.1261231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1dp_wb5/_remote_module_non_scriptable.py 2022-08-17T12:49:23.1478887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz3fcdt9 2022-08-17T12:49:23.1480003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz3fcdt9/_remote_module_non_scriptable.py 2022-08-17T12:49:23.2172978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmif71dpk 2022-08-17T12:49:23.2173844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmif71dpk/_remote_module_non_scriptable.py 2022-08-17T12:49:23.3435384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:23.3850097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:23.4107322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:23.4817465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:24.1455553Z ok (2.479s) 2022-08-17T12:49:24.1455851Z 2022-08-17T12:49:24.1456330Z ---------------------------------------------------------------------- 2022-08-17T12:49:24.1456766Z Ran 1 test in 2.479s 2022-08-17T12:49:24.1456946Z 2022-08-17T12:49:24.1457046Z OK 2022-08-17T12:49:24.1457179Z 2022-08-17T12:49:24.1457644Z Generating XML reports... 2022-08-17T12:49:24.1491402Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124921.xml 2022-08-17T12:49:25.4200455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:25.4200879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:25.4203697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:25.4204153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:25.5687740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq81twp4n 2022-08-17T12:49:25.5691208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq81twp4n/_remote_module_non_scriptable.py 2022-08-17T12:49:25.8306713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:25.8317276Z 2022-08-17T12:49:25.8317375Z Running tests... 2022-08-17T12:49:25.8317832Z ---------------------------------------------------------------------- 2022-08-17T12:49:26.0905415Z test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5381 2022-08-17T12:49:26.0933760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5382 2022-08-17T12:49:26.0962876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5383 2022-08-17T12:49:26.0993424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5384 2022-08-17T12:49:27.1719773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:27.1720474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:27.1724090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:27.1724755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:27.2037760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:27.2038453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:27.2041292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:27.2041688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:27.2147139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:27.2147778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:27.2151339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:27.2152013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:27.2357387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:27.2358041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:27.2361236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:27.2361879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:27.3337522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxopcjtgt 2022-08-17T12:49:27.3338493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxopcjtgt/_remote_module_non_scriptable.py 2022-08-17T12:49:27.3633886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpbwqnkr 2022-08-17T12:49:27.3635208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpbwqnkr/_remote_module_non_scriptable.py 2022-08-17T12:49:27.3748561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ykolu9_ 2022-08-17T12:49:27.3750489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ykolu9_/_remote_module_non_scriptable.py 2022-08-17T12:49:27.3931617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0jw1p3h 2022-08-17T12:49:27.3932797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0jw1p3h/_remote_module_non_scriptable.py 2022-08-17T12:49:27.5948206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:27.6269792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:27.6354918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:27.6511485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:27.8683439Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:49:27.8684377Z RuntimeError('Expected error') 2022-08-17T12:49:27.8684882Z Traceback (most recent call last): 2022-08-17T12:49:27.8686353Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:27.8687143Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:27.8688121Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:27.8688739Z return fn(*args, **kwargs) 2022-08-17T12:49:27.8689701Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:27.8690440Z raise RuntimeError("Expected error") 2022-08-17T12:49:27.8690943Z RuntimeError: Expected error 2022-08-17T12:49:27.8691210Z 2022-08-17T12:49:27.8881842Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:49:27.8882476Z RuntimeError('Expected error') 2022-08-17T12:49:27.8882869Z Traceback (most recent call last): 2022-08-17T12:49:27.8883593Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:27.8884200Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:27.8884924Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:27.8885423Z return fn(*args, **kwargs) 2022-08-17T12:49:27.8886365Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:27.8886939Z raise RuntimeError("Expected error") 2022-08-17T12:49:27.8887338Z RuntimeError: Expected error 2022-08-17T12:49:27.8887564Z 2022-08-17T12:49:27.8966156Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:49:27.8966938Z RuntimeError('Expected error') 2022-08-17T12:49:27.8967455Z Traceback (most recent call last): 2022-08-17T12:49:27.8968427Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:27.8969184Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:27.8970162Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:27.8970803Z return fn(*args, **kwargs) 2022-08-17T12:49:27.8971722Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:27.8972450Z raise RuntimeError("Expected error") 2022-08-17T12:49:27.8972948Z RuntimeError: Expected error 2022-08-17T12:49:27.8973235Z 2022-08-17T12:49:27.8997984Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:49:27.8998561Z RuntimeError('Expected error') 2022-08-17T12:49:27.8999061Z Traceback (most recent call last): 2022-08-17T12:49:27.9000301Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:27.9001041Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:27.9002014Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:27.9002660Z return fn(*args, **kwargs) 2022-08-17T12:49:27.9003616Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:27.9004331Z raise RuntimeError("Expected error") 2022-08-17T12:49:27.9004820Z RuntimeError: Expected error 2022-08-17T12:49:27.9005108Z 2022-08-17T12:49:28.3054702Z ok (2.473s) 2022-08-17T12:49:28.3054911Z 2022-08-17T12:49:28.3055237Z ---------------------------------------------------------------------- 2022-08-17T12:49:28.3055549Z Ran 1 test in 2.474s 2022-08-17T12:49:28.3055657Z 2022-08-17T12:49:28.3055722Z OK 2022-08-17T12:49:28.3055829Z 2022-08-17T12:49:28.3055921Z Generating XML reports... 2022-08-17T12:49:28.3089366Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124925.xml 2022-08-17T12:49:29.6163466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:29.6163884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:29.6166758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:29.6167134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:29.7695494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaim800v4 2022-08-17T12:49:29.7698891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaim800v4/_remote_module_non_scriptable.py 2022-08-17T12:49:30.0344170Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:30.0355145Z 2022-08-17T12:49:30.0355301Z Running tests... 2022-08-17T12:49:30.0355908Z ---------------------------------------------------------------------- 2022-08-17T12:49:30.2909973Z test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s) 2022-08-17T12:49:30.2910486Z 2022-08-17T12:49:30.2910717Z ---------------------------------------------------------------------- 2022-08-17T12:49:30.2910976Z Ran 1 test in 0.255s 2022-08-17T12:49:30.2911098Z 2022-08-17T12:49:30.2911173Z OK (skipped=1) 2022-08-17T12:49:30.2911287Z 2022-08-17T12:49:30.2911363Z Generating XML reports... 2022-08-17T12:49:30.2936611Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124930.xml 2022-08-17T12:49:31.5412373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:31.5412801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:31.5414680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:31.5415126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:31.6935477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr782k2e 2022-08-17T12:49:31.6938580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr782k2e/_remote_module_non_scriptable.py 2022-08-17T12:49:31.9567266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:31.9578168Z 2022-08-17T12:49:31.9578258Z Running tests... 2022-08-17T12:49:31.9578728Z ---------------------------------------------------------------------- 2022-08-17T12:49:32.2210491Z test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5614 2022-08-17T12:49:32.2240445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5615 2022-08-17T12:49:32.2269445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5616 2022-08-17T12:49:32.2300053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5617 2022-08-17T12:49:33.3386601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:33.3387313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:33.3390754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:33.3391446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:33.3402725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:33.3403373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:33.3407906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:33.3408576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:33.3741255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:33.3741985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:33.3746345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:33.3747041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:33.3866102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:33.3866684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:33.3870673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:33.3871340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:33.5015815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gmp2bze 2022-08-17T12:49:33.5017108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gmp2bze/_remote_module_non_scriptable.py 2022-08-17T12:49:33.5030590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4l_3quo 2022-08-17T12:49:33.5032945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4l_3quo/_remote_module_non_scriptable.py 2022-08-17T12:49:33.5355727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpls83wnh2 2022-08-17T12:49:33.5356670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpls83wnh2/_remote_module_non_scriptable.py 2022-08-17T12:49:33.5623959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhxelif6 2022-08-17T12:49:33.5625364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhxelif6/_remote_module_non_scriptable.py 2022-08-17T12:49:33.7655893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:33.7673479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:33.8010637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:33.8253808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:34.0085650Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:49:34.0086613Z RuntimeError('Expected error') 2022-08-17T12:49:34.0087120Z Traceback (most recent call last): 2022-08-17T12:49:34.0088076Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:34.0088899Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:34.0089867Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:34.0090514Z return fn(*args, **kwargs) 2022-08-17T12:49:34.0091454Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:34.0092209Z raise RuntimeError("Expected error") 2022-08-17T12:49:34.0092718Z RuntimeError: Expected error 2022-08-17T12:49:34.0093002Z 2022-08-17T12:49:34.0243717Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:49:34.0244562Z RuntimeError('Expected error') 2022-08-17T12:49:34.0245105Z Traceback (most recent call last): 2022-08-17T12:49:34.0246255Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:34.0247017Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:34.0248365Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:34.0249015Z return fn(*args, **kwargs) 2022-08-17T12:49:34.0249961Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:34.0250695Z raise RuntimeError("Expected error") 2022-08-17T12:49:34.0251196Z RuntimeError: Expected error 2022-08-17T12:49:34.0251482Z 2022-08-17T12:49:34.0283601Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:49:34.0284162Z RuntimeError('Expected error') 2022-08-17T12:49:34.0284568Z Traceback (most recent call last): 2022-08-17T12:49:34.0285307Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:34.0286077Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:34.0286855Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:34.0287360Z return fn(*args, **kwargs) 2022-08-17T12:49:34.0288101Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:34.0288597Z raise RuntimeError("Expected error") 2022-08-17T12:49:34.0288953Z RuntimeError: Expected error 2022-08-17T12:49:34.0289165Z 2022-08-17T12:49:34.0343144Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:49:34.0343822Z RuntimeError('Expected error') 2022-08-17T12:49:34.0344330Z Traceback (most recent call last): 2022-08-17T12:49:34.0345346Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:49:34.0346163Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:49:34.0347138Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper 2022-08-17T12:49:34.0347786Z return fn(*args, **kwargs) 2022-08-17T12:49:34.0348748Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 470, in async_raise_func 2022-08-17T12:49:34.0349466Z raise RuntimeError("Expected error") 2022-08-17T12:49:34.0349973Z RuntimeError: Expected error 2022-08-17T12:49:34.0350262Z 2022-08-17T12:49:34.4361469Z ok (2.478s) 2022-08-17T12:49:34.4361678Z 2022-08-17T12:49:34.4362171Z ---------------------------------------------------------------------- 2022-08-17T12:49:34.4362492Z Ran 1 test in 2.478s 2022-08-17T12:49:34.4362671Z 2022-08-17T12:49:34.4362747Z OK 2022-08-17T12:49:34.4362904Z 2022-08-17T12:49:34.4363766Z Generating XML reports... 2022-08-17T12:49:34.4396403Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124931.xml 2022-08-17T12:49:35.7480774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:35.7481473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:35.7484281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:35.7484887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:35.9007276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvuziqbf 2022-08-17T12:49:35.9011079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvuziqbf/_remote_module_non_scriptable.py 2022-08-17T12:49:36.1642942Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:36.1654499Z 2022-08-17T12:49:36.1654738Z Running tests... 2022-08-17T12:49:36.1655408Z ---------------------------------------------------------------------- 2022-08-17T12:49:36.4277355Z test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5837 2022-08-17T12:49:36.4306830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5838 2022-08-17T12:49:36.4336201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5839 2022-08-17T12:49:36.4369395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5840 2022-08-17T12:49:37.4616799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:37.4617517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:37.4620879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:37.4621553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:37.4912640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:37.4913323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:37.4915664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:37.4916342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:37.5354621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:37.5355376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:37.5357892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:37.5358658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:37.5388510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:37.5389308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:37.5393766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:37.5394536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:37.6237205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8aizj6we 2022-08-17T12:49:37.6238035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8aizj6we/_remote_module_non_scriptable.py 2022-08-17T12:49:37.6516604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdwev7_1 2022-08-17T12:49:37.6517416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdwev7_1/_remote_module_non_scriptable.py 2022-08-17T12:49:37.6971787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptoa2i2oj 2022-08-17T12:49:37.6973050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptoa2i2oj/_remote_module_non_scriptable.py 2022-08-17T12:49:37.7113844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszjwpdld 2022-08-17T12:49:37.7114621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszjwpdld/_remote_module_non_scriptable.py 2022-08-17T12:49:37.8858157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:37.9112082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:37.9609905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:37.9723334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:38.5426738Z ok (2.377s) 2022-08-17T12:49:38.5427186Z 2022-08-17T12:49:38.5427652Z ---------------------------------------------------------------------- 2022-08-17T12:49:38.5427907Z Ran 1 test in 2.377s 2022-08-17T12:49:38.5428029Z 2022-08-17T12:49:38.5428093Z OK 2022-08-17T12:49:38.5428189Z 2022-08-17T12:49:38.5428530Z Generating XML reports... 2022-08-17T12:49:38.5461166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124936.xml 2022-08-17T12:49:39.8596266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:39.8596662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:39.8599018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:39.8599397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:40.0131268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv36lveq 2022-08-17T12:49:40.0134931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv36lveq/_remote_module_non_scriptable.py 2022-08-17T12:49:40.2789087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:40.2800276Z 2022-08-17T12:49:40.2800627Z Running tests... 2022-08-17T12:49:40.2801021Z ---------------------------------------------------------------------- 2022-08-17T12:49:40.5496026Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6060 2022-08-17T12:49:40.5527087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6061 2022-08-17T12:49:40.5557524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6062 2022-08-17T12:49:40.5592155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6063 2022-08-17T12:49:41.5692913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:41.5693599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:41.5696450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:41.5697112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:41.5830603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:41.5831247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:41.5834510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:41.5835155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:41.6051383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:41.6051986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:41.6055050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:41.6055799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:41.6196179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:41.6196724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:41.6200167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:41.6200850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:41.7280471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoid7lh2n 2022-08-17T12:49:41.7281349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoid7lh2n/_remote_module_non_scriptable.py 2022-08-17T12:49:41.7642578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00h6zi6g 2022-08-17T12:49:41.7644001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00h6zi6g/_remote_module_non_scriptable.py 2022-08-17T12:49:41.7657391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfyw66_4 2022-08-17T12:49:41.7659925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfyw66_4/_remote_module_non_scriptable.py 2022-08-17T12:49:41.7849186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod67nafp 2022-08-17T12:49:41.7850273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod67nafp/_remote_module_non_scriptable.py 2022-08-17T12:49:41.9901221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:42.0224330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:42.0265375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:42.0445760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:42.6651218Z ok (2.385s) 2022-08-17T12:49:42.6651499Z 2022-08-17T12:49:42.6651857Z ---------------------------------------------------------------------- 2022-08-17T12:49:42.6652133Z Ran 1 test in 2.385s 2022-08-17T12:49:42.6652255Z 2022-08-17T12:49:42.6652322Z OK 2022-08-17T12:49:42.6652418Z 2022-08-17T12:49:42.6652517Z Generating XML reports... 2022-08-17T12:49:42.6685174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124940.xml 2022-08-17T12:49:43.9776545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:43.9776960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:43.9779229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:43.9779640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:44.1282402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthyeg_de 2022-08-17T12:49:44.1286096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthyeg_de/_remote_module_non_scriptable.py 2022-08-17T12:49:44.3879478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:44.3890616Z 2022-08-17T12:49:44.3890824Z Running tests... 2022-08-17T12:49:44.3891180Z ---------------------------------------------------------------------- 2022-08-17T12:49:44.6473969Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6283 2022-08-17T12:49:44.6501400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6284 2022-08-17T12:49:44.6530791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6285 2022-08-17T12:49:44.6561430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6286 2022-08-17T12:49:45.6876274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:45.6876885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:45.6879611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:45.6880294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:45.6943901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:45.6944586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:45.6948716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:45.6949666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:45.7295938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:45.7296575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:45.7298030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:45.7298606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:45.7362900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:45.7363469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:45.7366993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:45.7367537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:45.8468855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxh9i3zag 2022-08-17T12:49:45.8470189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxh9i3zag/_remote_module_non_scriptable.py 2022-08-17T12:49:45.8509625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r5baxq6 2022-08-17T12:49:45.8511467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r5baxq6/_remote_module_non_scriptable.py 2022-08-17T12:49:45.8880588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcy2_muy 2022-08-17T12:49:45.8881371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcy2_muy/_remote_module_non_scriptable.py 2022-08-17T12:49:45.9153761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk16ym2h 2022-08-17T12:49:45.9154585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk16ym2h/_remote_module_non_scriptable.py 2022-08-17T12:49:46.1088743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:46.1115673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:46.1507410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:46.1800325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:46.7620473Z ok (2.373s) 2022-08-17T12:49:46.7620729Z 2022-08-17T12:49:46.7621224Z ---------------------------------------------------------------------- 2022-08-17T12:49:46.7621679Z Ran 1 test in 2.373s 2022-08-17T12:49:46.7622068Z 2022-08-17T12:49:46.7622133Z OK 2022-08-17T12:49:46.7622231Z 2022-08-17T12:49:46.7622331Z Generating XML reports... 2022-08-17T12:49:46.7654870Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124944.xml 2022-08-17T12:49:48.0524256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:48.0524660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:48.0527443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:48.0527872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:48.2038772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppeuzli29 2022-08-17T12:49:48.2041360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppeuzli29/_remote_module_non_scriptable.py 2022-08-17T12:49:48.4660601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:48.4671411Z 2022-08-17T12:49:48.4671519Z Running tests... 2022-08-17T12:49:48.4672255Z ---------------------------------------------------------------------- 2022-08-17T12:49:48.7281429Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6506 2022-08-17T12:49:48.7309768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6507 2022-08-17T12:49:48.7338207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6508 2022-08-17T12:49:48.7368590Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6509 2022-08-17T12:49:49.7841426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:49.7842068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:49.7845268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:49.7846151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:49.7936571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:49.7937248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:49.7941367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:49.7942007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:49.8204802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:49.8205437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:49.8208297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:49.8208936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:49.8381699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:49.8382406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:49.8385136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:49.8385808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:49.9428973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl89mxmpo 2022-08-17T12:49:49.9429827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl89mxmpo/_remote_module_non_scriptable.py 2022-08-17T12:49:49.9533927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qc6ip54 2022-08-17T12:49:49.9535413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qc6ip54/_remote_module_non_scriptable.py 2022-08-17T12:49:49.9788577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a5xgxyh 2022-08-17T12:49:49.9789390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a5xgxyh/_remote_module_non_scriptable.py 2022-08-17T12:49:49.9962013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc6h74bh 2022-08-17T12:49:49.9963223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc6h74bh/_remote_module_non_scriptable.py 2022-08-17T12:49:50.2038899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:50.2164865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:50.2599705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:50.2637241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:50.8426065Z ok (2.375s) 2022-08-17T12:49:50.8426258Z 2022-08-17T12:49:50.8426584Z ---------------------------------------------------------------------- 2022-08-17T12:49:50.8427076Z Ran 1 test in 2.375s 2022-08-17T12:49:50.8427202Z 2022-08-17T12:49:50.8427269Z OK 2022-08-17T12:49:50.8427364Z 2022-08-17T12:49:50.8428682Z Generating XML reports... 2022-08-17T12:49:50.8462197Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124948.xml 2022-08-17T12:49:52.1311949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:52.1312373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:52.1313891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:52.1314284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:52.2816387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcpwcswu 2022-08-17T12:49:52.2819698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcpwcswu/_remote_module_non_scriptable.py 2022-08-17T12:49:52.5446450Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:52.5457343Z 2022-08-17T12:49:52.5457474Z Running tests... 2022-08-17T12:49:52.5457960Z ---------------------------------------------------------------------- 2022-08-17T12:49:52.8066610Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6729 2022-08-17T12:49:52.8094204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6730 2022-08-17T12:49:52.8123860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6731 2022-08-17T12:49:52.8155510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6732 2022-08-17T12:49:53.8671941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:53.8672637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:53.8675168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:53.8675829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:53.8745174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:53.8745844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:53.8749353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:53.8750374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:53.8909639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:53.8910311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:53.8912937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:53.8913646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:53.9312769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:53.9313388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:53.9316553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:53.9317179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:54.0269802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw7xhlfm 2022-08-17T12:49:54.0270900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw7xhlfm/_remote_module_non_scriptable.py 2022-08-17T12:49:54.0348625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuwqw9zs 2022-08-17T12:49:54.0350094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuwqw9zs/_remote_module_non_scriptable.py 2022-08-17T12:49:54.0496462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnpgh0m6 2022-08-17T12:49:54.0497491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnpgh0m6/_remote_module_non_scriptable.py 2022-08-17T12:49:54.0913463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fpktxla 2022-08-17T12:49:54.0914422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fpktxla/_remote_module_non_scriptable.py 2022-08-17T12:49:54.2881047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:54.3140440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:54.3195686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:54.3555167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:55.0216769Z ok (2.476s) 2022-08-17T12:49:55.0217095Z 2022-08-17T12:49:55.0217656Z ---------------------------------------------------------------------- 2022-08-17T12:49:55.0217960Z Ran 1 test in 2.476s 2022-08-17T12:49:55.0218086Z 2022-08-17T12:49:55.0218154Z OK 2022-08-17T12:49:55.0218237Z 2022-08-17T12:49:55.0218333Z Generating XML reports... 2022-08-17T12:49:55.0251608Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124952.xml 2022-08-17T12:49:56.3316460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:56.3316897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:56.3319471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:56.3319846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:56.4837360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaajl1n4 2022-08-17T12:49:56.4840283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaajl1n4/_remote_module_non_scriptable.py 2022-08-17T12:49:56.7490545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:49:56.7501223Z 2022-08-17T12:49:56.7501362Z Running tests... 2022-08-17T12:49:56.7502043Z ---------------------------------------------------------------------- 2022-08-17T12:49:57.0098766Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6952 2022-08-17T12:49:57.0128517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6953 2022-08-17T12:49:57.0156915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6954 2022-08-17T12:49:57.0186571Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6955 2022-08-17T12:49:58.0249198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:58.0249860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:58.0252595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:58.0253299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:58.0452980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:58.0453648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:58.0455671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:58.0456337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:58.0619310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:58.0620207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:58.0621087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:49:58.0621686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:49:58.0622955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:58.0623564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:58.0624300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:49:58.0625007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:49:58.1816961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse3q8szq 2022-08-17T12:49:58.1817885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse3q8szq/_remote_module_non_scriptable.py 2022-08-17T12:49:58.2041026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkswfm6d 2022-08-17T12:49:58.2041882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkswfm6d/_remote_module_non_scriptable.py 2022-08-17T12:49:58.2210359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59wi3lam 2022-08-17T12:49:58.2212039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5bltv8c 2022-08-17T12:49:58.2213141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59wi3lam/_remote_module_non_scriptable.py 2022-08-17T12:49:58.2213893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5bltv8c/_remote_module_non_scriptable.py 2022-08-17T12:49:58.4618716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:49:58.4629636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:49:58.4808852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:49:58.4821445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:49:59.1246071Z ok (2.374s) 2022-08-17T12:49:59.1246504Z 2022-08-17T12:49:59.1246921Z ---------------------------------------------------------------------- 2022-08-17T12:49:59.1247177Z Ran 1 test in 2.374s 2022-08-17T12:49:59.1247299Z 2022-08-17T12:49:59.1247362Z OK 2022-08-17T12:49:59.1247459Z 2022-08-17T12:49:59.1247618Z Generating XML reports... 2022-08-17T12:49:59.1279774Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124956.xml 2022-08-17T12:50:00.4049828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:00.4050224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:00.4052532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:00.4052900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:00.5678324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2uff7ah 2022-08-17T12:50:00.5681156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2uff7ah/_remote_module_non_scriptable.py 2022-08-17T12:50:00.8295543Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:00.8305719Z 2022-08-17T12:50:00.8306081Z Running tests... 2022-08-17T12:50:00.8306555Z ---------------------------------------------------------------------- 2022-08-17T12:50:01.0926214Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7175 2022-08-17T12:50:01.0954315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7176 2022-08-17T12:50:01.0982255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7177 2022-08-17T12:50:01.1014006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7178 2022-08-17T12:50:02.1099137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:02.1099787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:02.1102735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:02.1103361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:02.1180686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:02.1181341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:02.1184179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:02.1184809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:02.1549909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:02.1550629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:02.1551414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:02.1551947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:02.1552746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:02.1553360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:02.1555439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:02.1556127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:02.2700479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfts7u10k 2022-08-17T12:50:02.2701506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfts7u10k/_remote_module_non_scriptable.py 2022-08-17T12:50:02.2779327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37evdug4 2022-08-17T12:50:02.2780724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37evdug4/_remote_module_non_scriptable.py 2022-08-17T12:50:02.3144563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfoe0lpc 2022-08-17T12:50:02.3145322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhxklpbg 2022-08-17T12:50:02.3146042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfoe0lpc/_remote_module_non_scriptable.py 2022-08-17T12:50:02.3146779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhxklpbg/_remote_module_non_scriptable.py 2022-08-17T12:50:02.5316606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:02.5415259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:02.5765743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:02.5773174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:03.2073180Z ok (2.376s) 2022-08-17T12:50:03.2073379Z 2022-08-17T12:50:03.2073732Z ---------------------------------------------------------------------- 2022-08-17T12:50:03.2074045Z Ran 1 test in 2.377s 2022-08-17T12:50:03.2074167Z 2022-08-17T12:50:03.2074231Z OK 2022-08-17T12:50:03.2074327Z 2022-08-17T12:50:03.2074408Z Generating XML reports... 2022-08-17T12:50:03.2108557Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125000.xml 2022-08-17T12:50:04.5505199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:04.5505909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:04.5508485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:04.5508885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:04.7071002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o4wo16u 2022-08-17T12:50:04.7074708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o4wo16u/_remote_module_non_scriptable.py 2022-08-17T12:50:04.9712915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:04.9723246Z 2022-08-17T12:50:04.9723511Z Running tests... 2022-08-17T12:50:04.9724137Z ---------------------------------------------------------------------- 2022-08-17T12:50:05.2331919Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7398 2022-08-17T12:50:05.2360722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7399 2022-08-17T12:50:05.2390196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7400 2022-08-17T12:50:05.2422597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7401 2022-08-17T12:50:06.3138074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:06.3138689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:06.3176446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:06.3177145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:06.3184264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:06.3185536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:06.3190102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:06.3190749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:06.3621902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:06.3622574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:06.3626082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:06.3626708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:06.3658042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:06.3658601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:06.3663351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:06.3664082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:06.4749585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy6bhbgr 2022-08-17T12:50:06.4750519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy6bhbgr/_remote_module_non_scriptable.py 2022-08-17T12:50:06.4781849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjiytuqha 2022-08-17T12:50:06.4784127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjiytuqha/_remote_module_non_scriptable.py 2022-08-17T12:50:06.5206543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1sezp26j 2022-08-17T12:50:06.5207976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1sezp26j/_remote_module_non_scriptable.py 2022-08-17T12:50:06.5247539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppumgd4na 2022-08-17T12:50:06.5250089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppumgd4na/_remote_module_non_scriptable.py 2022-08-17T12:50:06.7356304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:06.7399370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:06.7880509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:06.7980679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:08.3496165Z ok (3.377s) 2022-08-17T12:50:08.3496436Z 2022-08-17T12:50:08.3496977Z ---------------------------------------------------------------------- 2022-08-17T12:50:08.3497242Z Ran 1 test in 3.377s 2022-08-17T12:50:08.3497366Z 2022-08-17T12:50:08.3497430Z OK 2022-08-17T12:50:08.3497544Z 2022-08-17T12:50:08.3497640Z Generating XML reports... 2022-08-17T12:50:08.3532786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125004.xml 2022-08-17T12:50:09.6505480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:09.6505936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:09.6508720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:09.6509093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:09.8033473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptatftz15 2022-08-17T12:50:09.8036840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptatftz15/_remote_module_non_scriptable.py 2022-08-17T12:50:10.0693575Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:10.0703626Z 2022-08-17T12:50:10.0703707Z Running tests... 2022-08-17T12:50:10.0704445Z ---------------------------------------------------------------------- 2022-08-17T12:50:10.3338386Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7621 2022-08-17T12:50:10.3368030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7622 2022-08-17T12:50:10.3397968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7623 2022-08-17T12:50:10.3431130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7624 2022-08-17T12:50:11.4205584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:11.4206473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:11.4209250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:11.4209888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:11.4465248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:11.4465999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:11.4468428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:11.4469107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:11.4694339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:11.4694917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:11.4697936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:11.4698485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:11.4742315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:11.4742861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:11.4747094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:11.4747735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:11.5800215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgjyc2sn 2022-08-17T12:50:11.5801059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgjyc2sn/_remote_module_non_scriptable.py 2022-08-17T12:50:11.6053327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3z1z3ty7 2022-08-17T12:50:11.6054173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3z1z3ty7/_remote_module_non_scriptable.py 2022-08-17T12:50:11.6326564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w9erec1 2022-08-17T12:50:11.6327785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w9erec1/_remote_module_non_scriptable.py 2022-08-17T12:50:11.6328564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfj8owlq 2022-08-17T12:50:11.6330621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfj8owlq/_remote_module_non_scriptable.py 2022-08-17T12:50:11.8428536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:11.8672942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:11.8920073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:11.8968737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:13.5508231Z ok (3.480s) 2022-08-17T12:50:13.5508473Z 2022-08-17T12:50:13.5508954Z ---------------------------------------------------------------------- 2022-08-17T12:50:13.5509333Z Ran 1 test in 3.480s 2022-08-17T12:50:13.5509535Z 2022-08-17T12:50:13.5509642Z OK 2022-08-17T12:50:13.5509791Z 2022-08-17T12:50:13.5509933Z Generating XML reports... 2022-08-17T12:50:13.5543441Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125010.xml 2022-08-17T12:50:14.8405008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:14.8405697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:14.8409451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:14.8410047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:14.9922958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqbqln8p 2022-08-17T12:50:14.9927541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqbqln8p/_remote_module_non_scriptable.py 2022-08-17T12:50:15.2607161Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:15.2618888Z 2022-08-17T12:50:15.2619215Z Running tests... 2022-08-17T12:50:15.2619867Z ---------------------------------------------------------------------- 2022-08-17T12:50:15.5263329Z test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7844 2022-08-17T12:50:15.5290488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7845 2022-08-17T12:50:15.5318696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7846 2022-08-17T12:50:15.5348569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7847 2022-08-17T12:50:16.6054750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:16.6055448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:16.6058038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:16.6058700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:16.6230033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:16.6230717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:16.6234246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:16.6234953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:16.6748034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:16.6748788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:16.6752370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:16.6752985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:16.6897170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:16.6897875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:16.6901182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:16.6901800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:16.7627114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4arp6os6 2022-08-17T12:50:16.7627919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4arp6os6/_remote_module_non_scriptable.py 2022-08-17T12:50:16.7838111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17n7fi74 2022-08-17T12:50:16.7839391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17n7fi74/_remote_module_non_scriptable.py 2022-08-17T12:50:16.8350678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaberyfv 2022-08-17T12:50:16.8351493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaberyfv/_remote_module_non_scriptable.py 2022-08-17T12:50:16.8493387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5u7wrak 2022-08-17T12:50:16.8494555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5u7wrak/_remote_module_non_scriptable.py 2022-08-17T12:50:17.0225618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:17.0470751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:17.0966130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:17.1118933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:17.7409877Z ok (2.479s) 2022-08-17T12:50:17.7410123Z 2022-08-17T12:50:17.7410587Z ---------------------------------------------------------------------- 2022-08-17T12:50:17.7410842Z Ran 1 test in 2.479s 2022-08-17T12:50:17.7411014Z 2022-08-17T12:50:17.7411111Z OK 2022-08-17T12:50:17.7411206Z 2022-08-17T12:50:17.7411304Z Generating XML reports... 2022-08-17T12:50:17.7444663Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125015.xml 2022-08-17T12:50:19.0505300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:19.0505733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:19.0508443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:19.0508828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:19.2029572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8se2h8hv 2022-08-17T12:50:19.2032746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8se2h8hv/_remote_module_non_scriptable.py 2022-08-17T12:50:19.4666121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:19.4676550Z 2022-08-17T12:50:19.4676754Z Running tests... 2022-08-17T12:50:19.4677097Z ---------------------------------------------------------------------- 2022-08-17T12:50:19.7303969Z test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8067 2022-08-17T12:50:19.7334239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8068 2022-08-17T12:50:19.7364053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8069 2022-08-17T12:50:19.7396141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8070 2022-08-17T12:50:20.7474825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:20.7475474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:20.7477995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:20.7478615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:20.7891694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:20.7892320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:20.7895483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:20.7896099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:20.7946516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:20.7947223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:20.7950774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:20.7951465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:20.7955382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:20.7955991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:20.7960623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:20.7961492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:20.9063801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0kwi3c7 2022-08-17T12:50:20.9064626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0kwi3c7/_remote_module_non_scriptable.py 2022-08-17T12:50:20.9498000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w9stej6 2022-08-17T12:50:20.9498772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w9stej6/_remote_module_non_scriptable.py 2022-08-17T12:50:20.9546529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_230luq 2022-08-17T12:50:20.9547386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_230luq/_remote_module_non_scriptable.py 2022-08-17T12:50:20.9565595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgm1gh9l 2022-08-17T12:50:20.9567883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgm1gh9l/_remote_module_non_scriptable.py 2022-08-17T12:50:21.1688026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:21.2135335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:21.2154270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:21.2201638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:21.8454286Z ok (2.377s) 2022-08-17T12:50:21.8454511Z 2022-08-17T12:50:21.8454883Z ---------------------------------------------------------------------- 2022-08-17T12:50:21.8455172Z Ran 1 test in 2.378s 2022-08-17T12:50:21.8455295Z 2022-08-17T12:50:21.8455360Z OK 2022-08-17T12:50:21.8455458Z 2022-08-17T12:50:21.8455557Z Generating XML reports... 2022-08-17T12:50:21.8492293Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125019.xml 2022-08-17T12:50:23.1588732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:23.1589354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:23.1591501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:23.1592081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:23.3114150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38y5xw15 2022-08-17T12:50:23.3118024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38y5xw15/_remote_module_non_scriptable.py 2022-08-17T12:50:23.5723603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:23.5735114Z 2022-08-17T12:50:23.5735451Z Running tests... 2022-08-17T12:50:23.5736155Z ---------------------------------------------------------------------- 2022-08-17T12:50:23.8408266Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8290 2022-08-17T12:50:23.8437227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8291 2022-08-17T12:50:23.8466489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8292 2022-08-17T12:50:23.8497852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8293 2022-08-17T12:50:24.8832043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:24.8832690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:24.8835396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:24.8836260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:24.9058565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:24.9059209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:24.9062671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:24.9063299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:24.9139409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:24.9140085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:24.9144522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:24.9145277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:24.9153142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:24.9153766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:24.9156897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:24.9157549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:25.0415041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxf82242 2022-08-17T12:50:25.0416256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxf82242/_remote_module_non_scriptable.py 2022-08-17T12:50:25.0645240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vzlz_5j 2022-08-17T12:50:25.0646850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vzlz_5j/_remote_module_non_scriptable.py 2022-08-17T12:50:25.0711473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyszmob6j 2022-08-17T12:50:25.0713422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyszmob6j/_remote_module_non_scriptable.py 2022-08-17T12:50:25.0755371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9f15y6e 2022-08-17T12:50:25.0757367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9f15y6e/_remote_module_non_scriptable.py 2022-08-17T12:50:25.3008225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:25.3226388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:25.3291600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:25.3484127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:25.6550883Z ok (2.081s) 2022-08-17T12:50:25.6551154Z 2022-08-17T12:50:25.6551684Z ---------------------------------------------------------------------- 2022-08-17T12:50:25.6552150Z Ran 1 test in 2.081s 2022-08-17T12:50:25.6552364Z 2022-08-17T12:50:25.6552468Z OK 2022-08-17T12:50:25.6552642Z 2022-08-17T12:50:25.6552806Z Generating XML reports... 2022-08-17T12:50:25.6585592Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125023.xml 2022-08-17T12:50:26.9442574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:26.9443272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:26.9446773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:26.9447403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:27.0971247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoj9tlzz 2022-08-17T12:50:27.0974583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoj9tlzz/_remote_module_non_scriptable.py 2022-08-17T12:50:27.3610156Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:27.3620594Z 2022-08-17T12:50:27.3620705Z Running tests... 2022-08-17T12:50:27.3621308Z ---------------------------------------------------------------------- 2022-08-17T12:50:27.6256392Z test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8345 2022-08-17T12:50:27.6285366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8346 2022-08-17T12:50:27.6313787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8347 2022-08-17T12:50:27.6345272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8348 2022-08-17T12:50:28.6870945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:28.6871583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:28.6875092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:28.6875756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:28.6939909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:28.6940560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:28.6945525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:28.6946164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:28.7028503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:28.7029164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:28.7032181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:28.7032800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:28.7502851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:28.7503668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:28.7507097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:28.7508085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:28.8609583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzfkcl7z 2022-08-17T12:50:28.8610507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzfkcl7z/_remote_module_non_scriptable.py 2022-08-17T12:50:28.8663351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7evo2f9h 2022-08-17T12:50:28.8664995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7evo2f9h/_remote_module_non_scriptable.py 2022-08-17T12:50:28.8676132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_yfsnnz 2022-08-17T12:50:28.8678076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_yfsnnz/_remote_module_non_scriptable.py 2022-08-17T12:50:28.9127318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyhb6h6g 2022-08-17T12:50:28.9128511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyhb6h6g/_remote_module_non_scriptable.py 2022-08-17T12:50:29.1271117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:29.1317402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:29.1365034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:29.1807551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:29.8407672Z ok (2.478s) 2022-08-17T12:50:29.8407897Z 2022-08-17T12:50:29.8408371Z ---------------------------------------------------------------------- 2022-08-17T12:50:29.8408641Z Ran 1 test in 2.479s 2022-08-17T12:50:29.8408767Z 2022-08-17T12:50:29.8408829Z OK 2022-08-17T12:50:29.8409002Z 2022-08-17T12:50:29.8409097Z Generating XML reports... 2022-08-17T12:50:29.8442013Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125027.xml 2022-08-17T12:50:31.1210660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:31.1211060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:31.1213636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:31.1214017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:31.2743814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ai3vtqe 2022-08-17T12:50:31.2747475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ai3vtqe/_remote_module_non_scriptable.py 2022-08-17T12:50:31.5412825Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:31.5423882Z 2022-08-17T12:50:31.5424183Z Running tests... 2022-08-17T12:50:31.5424830Z ---------------------------------------------------------------------- 2022-08-17T12:50:31.8066462Z test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8568 2022-08-17T12:50:31.8095405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8569 2022-08-17T12:50:31.8124282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8570 2022-08-17T12:50:31.8156591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8571 2022-08-17T12:50:32.8855524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:32.8856161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:32.8859035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:32.8860010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:32.8964188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:32.8964898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:32.8968057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:32.8968711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:32.9257298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:32.9258029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:32.9260474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:32.9261121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:32.9486160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:32.9486705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:32.9490272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:32.9491033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:33.0476688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw25_xtfn 2022-08-17T12:50:33.0477498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw25_xtfn/_remote_module_non_scriptable.py 2022-08-17T12:50:33.0567122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3900wt7c 2022-08-17T12:50:33.0568282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3900wt7c/_remote_module_non_scriptable.py 2022-08-17T12:50:33.0913299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeek7jv5j 2022-08-17T12:50:33.0914053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeek7jv5j/_remote_module_non_scriptable.py 2022-08-17T12:50:33.1165489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph98negu 2022-08-17T12:50:33.1166464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph98negu/_remote_module_non_scriptable.py 2022-08-17T12:50:33.3113866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:33.3192268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:33.3559602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:33.3825997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:34.0216835Z ok (2.479s) 2022-08-17T12:50:34.0217117Z 2022-08-17T12:50:34.0217662Z ---------------------------------------------------------------------- 2022-08-17T12:50:34.0218117Z Ran 1 test in 2.479s 2022-08-17T12:50:34.0218340Z 2022-08-17T12:50:34.0218469Z OK 2022-08-17T12:50:34.0218595Z 2022-08-17T12:50:34.0218694Z Generating XML reports... 2022-08-17T12:50:34.0252629Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125031.xml 2022-08-17T12:50:35.3279828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:35.3280228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:35.3282484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:35.3282848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:35.4783412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_u_l_cp 2022-08-17T12:50:35.4787009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_u_l_cp/_remote_module_non_scriptable.py 2022-08-17T12:50:35.7380593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:35.7391812Z 2022-08-17T12:50:35.7391961Z Running tests... 2022-08-17T12:50:35.7392470Z ---------------------------------------------------------------------- 2022-08-17T12:50:35.7403630Z test_call_method_on_rref (__main__.TensorPipeRpcTest) 2022-08-17T12:50:35.9991073Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8791 2022-08-17T12:50:36.0019793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8792 2022-08-17T12:50:36.0050269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8793 2022-08-17T12:50:36.0081605Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8794 2022-08-17T12:50:37.0067884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:37.0068494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:37.0071715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:37.0072370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:37.0355224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:37.0355898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:37.0358295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:37.0358937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:37.0396072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:37.0396723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:37.0400592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:37.0401278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:37.0531801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:37.0532472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:37.0536059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:37.0536754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:37.1689917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrkin8iv 2022-08-17T12:50:37.1690768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrkin8iv/_remote_module_non_scriptable.py 2022-08-17T12:50:37.2005360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf_mdzaa 2022-08-17T12:50:37.2007115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf_mdzaa/_remote_module_non_scriptable.py 2022-08-17T12:50:37.2127236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xdq6cpd 2022-08-17T12:50:37.2128260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xdq6cpd/_remote_module_non_scriptable.py 2022-08-17T12:50:37.2140573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv4d8hmm 2022-08-17T12:50:37.2142100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv4d8hmm/_remote_module_non_scriptable.py 2022-08-17T12:50:37.4281881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:37.4628914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:37.4726614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:37.4748673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:38.2142848Z ok (2.475s) 2022-08-17T12:50:38.2143133Z 2022-08-17T12:50:38.2143625Z ---------------------------------------------------------------------- 2022-08-17T12:50:38.2143959Z Ran 1 test in 2.475s 2022-08-17T12:50:38.2144190Z 2022-08-17T12:50:38.2144310Z OK 2022-08-17T12:50:38.2144486Z 2022-08-17T12:50:38.2144635Z Generating XML reports... 2022-08-17T12:50:38.2177659Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125035.xml 2022-08-17T12:50:39.5333547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:39.5334031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:39.5335631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:39.5336248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:39.6859632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn92wc_go 2022-08-17T12:50:39.6863321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn92wc_go/_remote_module_non_scriptable.py 2022-08-17T12:50:39.9511534Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:39.9522587Z 2022-08-17T12:50:39.9522681Z Running tests... 2022-08-17T12:50:39.9523104Z ---------------------------------------------------------------------- 2022-08-17T12:50:40.2205666Z test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9014 2022-08-17T12:50:40.2233208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9015 2022-08-17T12:50:40.2265852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9016 2022-08-17T12:50:40.2296485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9017 2022-08-17T12:50:41.2529550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:41.2530195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:41.2533206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:41.2533799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:41.2594281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:41.2594982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:41.2595811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:41.2596356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:41.2598636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:41.2599219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:41.2625634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:41.2626291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:41.2630074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:41.2631020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:41.2639991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:41.2640655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:41.4254562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oterzb0 2022-08-17T12:50:41.4255326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd7ckr0_ 2022-08-17T12:50:41.4256028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oterzb0/_remote_module_non_scriptable.py 2022-08-17T12:50:41.4256773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd7ckr0_/_remote_module_non_scriptable.py 2022-08-17T12:50:41.4259614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxmv5yeu 2022-08-17T12:50:41.4262441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxmv5yeu/_remote_module_non_scriptable.py 2022-08-17T12:50:41.4275495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjinrwn3 2022-08-17T12:50:41.4277698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjinrwn3/_remote_module_non_scriptable.py 2022-08-17T12:50:41.6873674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:41.6874410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:41.6887413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:41.6888704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:42.3355697Z ok (2.383s) 2022-08-17T12:50:42.3355896Z 2022-08-17T12:50:42.3356422Z ---------------------------------------------------------------------- 2022-08-17T12:50:42.3356908Z Ran 1 test in 2.383s 2022-08-17T12:50:42.3357022Z 2022-08-17T12:50:42.3357094Z OK 2022-08-17T12:50:42.3357191Z 2022-08-17T12:50:42.3357287Z Generating XML reports... 2022-08-17T12:50:42.3390706Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125039.xml 2022-08-17T12:50:43.6493975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:43.6494461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:43.6497087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:43.6497468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:43.8005365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t9bpo5j 2022-08-17T12:50:43.8009250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t9bpo5j/_remote_module_non_scriptable.py 2022-08-17T12:50:44.0646422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:44.0658044Z 2022-08-17T12:50:44.0658325Z Running tests... 2022-08-17T12:50:44.0658969Z ---------------------------------------------------------------------- 2022-08-17T12:50:44.3335502Z test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9237 2022-08-17T12:50:44.3365663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9238 2022-08-17T12:50:44.3404276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9239 2022-08-17T12:50:44.3443460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9240 2022-08-17T12:50:45.3547376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:45.3548328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:45.3550881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:45.3551497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:45.3737621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:45.3738279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:45.3740753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:45.3741402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:45.3791359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:45.3792068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:45.3796076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:45.3796670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:45.3980805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:45.3981514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:45.3984830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:45.3985524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:45.5133325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv8zu1nn 2022-08-17T12:50:45.5134305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv8zu1nn/_remote_module_non_scriptable.py 2022-08-17T12:50:45.5407300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrgr8maj 2022-08-17T12:50:45.5408288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrgr8maj/_remote_module_non_scriptable.py 2022-08-17T12:50:45.5557545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wf507bl 2022-08-17T12:50:45.5558435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wf507bl/_remote_module_non_scriptable.py 2022-08-17T12:50:45.5559106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07q4n_iu 2022-08-17T12:50:45.5561480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07q4n_iu/_remote_module_non_scriptable.py 2022-08-17T12:50:45.7722979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:45.8023491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:45.8152556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:45.8158564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:46.4501039Z ok (2.384s) 2022-08-17T12:50:46.4501283Z 2022-08-17T12:50:46.4501974Z ---------------------------------------------------------------------- 2022-08-17T12:50:46.4502449Z Ran 1 test in 2.384s 2022-08-17T12:50:46.4502672Z 2022-08-17T12:50:46.4502787Z OK 2022-08-17T12:50:46.4502953Z 2022-08-17T12:50:46.4503045Z Generating XML reports... 2022-08-17T12:50:46.4537692Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125044.xml 2022-08-17T12:50:47.7551008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:47.7551401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:47.7553883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:47.7554595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:47.9064993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_kitwv6 2022-08-17T12:50:47.9068037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_kitwv6/_remote_module_non_scriptable.py 2022-08-17T12:50:48.1672967Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:48.1683653Z 2022-08-17T12:50:48.1683794Z Running tests... 2022-08-17T12:50:48.1684421Z ---------------------------------------------------------------------- 2022-08-17T12:50:48.4261047Z test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9460 2022-08-17T12:50:48.4288806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9461 2022-08-17T12:50:48.4318122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9462 2022-08-17T12:50:48.4350853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9463 2022-08-17T12:50:49.5099654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:49.5100623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:49.5103497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:49.5104169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:49.5201754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:49.5202417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:49.5205803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:49.5206702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:49.5428101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:49.5428792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:49.5431429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:49.5432097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:49.5751145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:49.5751700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:49.5755021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:49.5755610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:49.6718292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr99n2zp 2022-08-17T12:50:49.6719106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr99n2zp/_remote_module_non_scriptable.py 2022-08-17T12:50:49.6802063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf4a9rvk 2022-08-17T12:50:49.6802900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf4a9rvk/_remote_module_non_scriptable.py 2022-08-17T12:50:49.7069500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7jq7ntv 2022-08-17T12:50:49.7071162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7jq7ntv/_remote_module_non_scriptable.py 2022-08-17T12:50:49.7401499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgiggp0q 2022-08-17T12:50:49.7402347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgiggp0q/_remote_module_non_scriptable.py 2022-08-17T12:50:49.9329196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:49.9405468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:49.9701618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:50.0037460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:50.6411728Z ok (2.472s) 2022-08-17T12:50:50.6411923Z 2022-08-17T12:50:50.6412246Z ---------------------------------------------------------------------- 2022-08-17T12:50:50.6412532Z Ran 1 test in 2.473s 2022-08-17T12:50:50.6412653Z 2022-08-17T12:50:50.6412716Z OK 2022-08-17T12:50:50.6412818Z 2022-08-17T12:50:50.6412900Z Generating XML reports... 2022-08-17T12:50:50.6445254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125048.xml 2022-08-17T12:50:51.9558808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:51.9559260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:51.9562325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:51.9562791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:52.1068821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvjsn6mw 2022-08-17T12:50:52.1072270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvjsn6mw/_remote_module_non_scriptable.py 2022-08-17T12:50:52.3722209Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:52.3733282Z 2022-08-17T12:50:52.3733382Z Running tests... 2022-08-17T12:50:52.3733871Z ---------------------------------------------------------------------- 2022-08-17T12:50:52.6403407Z test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9683 2022-08-17T12:50:52.6432654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9684 2022-08-17T12:50:52.6460896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9685 2022-08-17T12:50:52.6491715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9686 2022-08-17T12:50:53.7152026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:53.7152677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:53.7155641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:53.7156279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:53.7591606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:53.7592255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:53.7595261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:53.7595867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:53.7641155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:53.7641712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:53.7645733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:53.7646608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:53.7663628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:53.7664261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:53.7669253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:53.7669893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:53.8745130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_oyp0l3 2022-08-17T12:50:53.8746759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_oyp0l3/_remote_module_non_scriptable.py 2022-08-17T12:50:53.9189532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcfh0_8s 2022-08-17T12:50:53.9190390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcfh0_8s/_remote_module_non_scriptable.py 2022-08-17T12:50:53.9251005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy1on82n 2022-08-17T12:50:53.9251809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kbcqdrz 2022-08-17T12:50:53.9252533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy1on82n/_remote_module_non_scriptable.py 2022-08-17T12:50:53.9253501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kbcqdrz/_remote_module_non_scriptable.py 2022-08-17T12:50:54.1363856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:54.1816202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:54.1881432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:54.1904053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:54.4082971Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:50:54.4084124Z ValueError('Expected error') 2022-08-17T12:50:54.4084594Z Traceback (most recent call last): 2022-08-17T12:50:54.4085568Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:50:54.4086513Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:50:54.4087593Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:50:54.4088283Z raise ValueError(expected_err) 2022-08-17T12:50:54.4088762Z ValueError: Expected error 2022-08-17T12:50:54.4089044Z 2022-08-17T12:50:54.4280483Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:50:54.4281673Z ValueError('Expected error') 2022-08-17T12:50:54.4282664Z Traceback (most recent call last): 2022-08-17T12:50:54.4288736Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:50:54.4289372Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:50:54.4290206Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:50:54.4290735Z raise ValueError(expected_err) 2022-08-17T12:50:54.4291126Z ValueError: Expected error 2022-08-17T12:50:54.4291350Z 2022-08-17T12:50:54.4391909Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:50:54.4392657Z ValueError('Expected error') 2022-08-17T12:50:54.4393124Z Traceback (most recent call last): 2022-08-17T12:50:54.4394061Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:50:54.4394839Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:50:54.4395864Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:50:54.4396575Z raise ValueError(expected_err) 2022-08-17T12:50:54.4397054Z ValueError: Expected error 2022-08-17T12:50:54.4397639Z 2022-08-17T12:50:54.4452818Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:50:54.4453435Z ValueError('Expected error') 2022-08-17T12:50:54.4453922Z Traceback (most recent call last): 2022-08-17T12:50:54.4454852Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:50:54.4455603Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:50:54.4456636Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:50:54.4457340Z raise ValueError(expected_err) 2022-08-17T12:50:54.4457792Z ValueError: Expected error 2022-08-17T12:50:54.4458073Z 2022-08-17T12:50:54.8552740Z ok (2.482s) 2022-08-17T12:50:54.8553019Z 2022-08-17T12:50:54.8553436Z ---------------------------------------------------------------------- 2022-08-17T12:50:54.8553701Z Ran 1 test in 2.482s 2022-08-17T12:50:54.8553810Z 2022-08-17T12:50:54.8553891Z OK 2022-08-17T12:50:54.8553986Z 2022-08-17T12:50:54.8554081Z Generating XML reports... 2022-08-17T12:50:54.8587796Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125052.xml 2022-08-17T12:50:56.1608998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:56.1609390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:56.1611993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:56.1612514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:56.3129797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzebx7ksv 2022-08-17T12:50:56.3133063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzebx7ksv/_remote_module_non_scriptable.py 2022-08-17T12:50:56.5817992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:50:56.5828665Z 2022-08-17T12:50:56.5828799Z Running tests... 2022-08-17T12:50:56.5829414Z ---------------------------------------------------------------------- 2022-08-17T12:50:56.8547366Z test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9906 2022-08-17T12:50:56.8576830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9907 2022-08-17T12:50:56.8608259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9908 2022-08-17T12:50:56.8637454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9909 2022-08-17T12:50:57.8975720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:57.8976414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:57.8979588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:57.8980135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:57.9015547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:57.9016206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:57.9020108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:57.9020605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:57.9021082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:57.9021534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:57.9024653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:57.9025396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:57.9086560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:50:57.9087239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:50:57.9090099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:50:57.9090713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:50:58.0601594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kkoys8_ 2022-08-17T12:50:58.0602429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kkoys8_/_remote_module_non_scriptable.py 2022-08-17T12:50:58.0620009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_qb4w70 2022-08-17T12:50:58.0621972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_qb4w70/_remote_module_non_scriptable.py 2022-08-17T12:50:58.0625054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkblbcayz 2022-08-17T12:50:58.0628349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkblbcayz/_remote_module_non_scriptable.py 2022-08-17T12:50:58.0668115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat5zh6v1 2022-08-17T12:50:58.0670107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat5zh6v1/_remote_module_non_scriptable.py 2022-08-17T12:50:58.3198417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:50:58.3223610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:50:58.3248175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:50:58.3273581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:50:58.9694302Z ok (2.386s) 2022-08-17T12:50:58.9694824Z 2022-08-17T12:50:58.9695313Z ---------------------------------------------------------------------- 2022-08-17T12:50:58.9695636Z Ran 1 test in 2.386s 2022-08-17T12:50:58.9695796Z 2022-08-17T12:50:58.9695863Z OK 2022-08-17T12:50:58.9695960Z 2022-08-17T12:50:58.9696057Z Generating XML reports... 2022-08-17T12:50:58.9729026Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125056.xml 2022-08-17T12:51:00.2682241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:00.2682632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:00.2685025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:00.2685549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:00.4198717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrtz8t0o 2022-08-17T12:51:00.4201817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrtz8t0o/_remote_module_non_scriptable.py 2022-08-17T12:51:00.6842412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:00.6853585Z 2022-08-17T12:51:00.6853727Z Running tests... 2022-08-17T12:51:00.6854334Z ---------------------------------------------------------------------- 2022-08-17T12:51:00.9511117Z test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10129 2022-08-17T12:51:00.9540647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10130 2022-08-17T12:51:00.9569000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10131 2022-08-17T12:51:00.9601194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10132 2022-08-17T12:51:02.0673809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:02.0674509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:02.0677391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:02.0678049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:02.0885039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:02.0885666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:02.0888780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:02.0889456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:02.0905880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:02.0906570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:02.0910878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:02.0911589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:02.0967818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:02.0968457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:02.0971942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:02.0972625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:02.2249952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pjgi2rc 2022-08-17T12:51:02.2250791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pjgi2rc/_remote_module_non_scriptable.py 2022-08-17T12:51:02.2493259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o_07wsc 2022-08-17T12:51:02.2494130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o_07wsc/_remote_module_non_scriptable.py 2022-08-17T12:51:02.2515247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdvf6vzq 2022-08-17T12:51:02.2517590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdvf6vzq/_remote_module_non_scriptable.py 2022-08-17T12:51:02.2559264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply8kqfan 2022-08-17T12:51:02.2561462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply8kqfan/_remote_module_non_scriptable.py 2022-08-17T12:51:02.4845637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:02.5109987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:02.5121699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:02.5174592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:02.7356103Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:51:02.7356887Z ValueError('Expected error') 2022-08-17T12:51:02.7357364Z Traceback (most recent call last): 2022-08-17T12:51:02.7358324Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:51:02.7359117Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:51:02.7360166Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:51:02.7361235Z raise ValueError(expected_err) 2022-08-17T12:51:02.7361717Z ValueError: Expected error 2022-08-17T12:51:02.7362001Z 2022-08-17T12:51:02.7528687Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:51:02.7529493Z ValueError('Expected error') 2022-08-17T12:51:02.7530020Z Traceback (most recent call last): 2022-08-17T12:51:02.7530973Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:51:02.7531755Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:51:02.7532798Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:51:02.7533510Z raise ValueError(expected_err) 2022-08-17T12:51:02.7533968Z ValueError: Expected error 2022-08-17T12:51:02.7534254Z 2022-08-17T12:51:02.7557709Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:51:02.7558484Z ValueError('Expected error') 2022-08-17T12:51:02.7559042Z Traceback (most recent call last): 2022-08-17T12:51:02.7559969Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:51:02.7560758Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:51:02.7561994Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:51:02.7562525Z raise ValueError(expected_err) 2022-08-17T12:51:02.7562914Z ValueError: Expected error 2022-08-17T12:51:02.7563140Z 2022-08-17T12:51:02.7620492Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:51:02.7621173Z ValueError('Expected error') 2022-08-17T12:51:02.7621684Z Traceback (most recent call last): 2022-08-17T12:51:02.7622630Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:51:02.7623411Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:51:02.7624462Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:51:02.7625250Z raise ValueError(expected_err) 2022-08-17T12:51:02.7625728Z ValueError: Expected error 2022-08-17T12:51:02.7626011Z 2022-08-17T12:51:03.1660081Z ok (2.480s) 2022-08-17T12:51:03.1660342Z 2022-08-17T12:51:03.1660857Z ---------------------------------------------------------------------- 2022-08-17T12:51:03.1661184Z Ran 1 test in 2.481s 2022-08-17T12:51:03.1661316Z 2022-08-17T12:51:03.1661380Z OK 2022-08-17T12:51:03.1661477Z 2022-08-17T12:51:03.1661574Z Generating XML reports... 2022-08-17T12:51:03.1694506Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125100.xml 2022-08-17T12:51:04.4701889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:04.4702300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:04.4704185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:04.4704547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:04.6221719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n0hlezk 2022-08-17T12:51:04.6225633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n0hlezk/_remote_module_non_scriptable.py 2022-08-17T12:51:04.8844984Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:04.8855855Z 2022-08-17T12:51:04.8855937Z Running tests... 2022-08-17T12:51:04.8856918Z ---------------------------------------------------------------------- 2022-08-17T12:51:05.1499454Z test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10352 2022-08-17T12:51:05.1528282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10353 2022-08-17T12:51:05.1565139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10354 2022-08-17T12:51:05.1603271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10355 2022-08-17T12:51:06.1300530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:06.1301209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:06.1304759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:06.1305442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:06.1583280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:06.1583988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:06.1587405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:06.1588110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:06.1960183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:06.1960935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:06.1965320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:06.1966187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:06.2087377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:06.2087801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:06.2090626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:06.2091314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:06.2908825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw9lk366 2022-08-17T12:51:06.2909910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw9lk366/_remote_module_non_scriptable.py 2022-08-17T12:51:06.3390705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5da4_ksd 2022-08-17T12:51:06.3391559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5da4_ksd/_remote_module_non_scriptable.py 2022-08-17T12:51:06.3590579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk487924 2022-08-17T12:51:06.3591731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk487924/_remote_module_non_scriptable.py 2022-08-17T12:51:06.3702567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhrytb2w 2022-08-17T12:51:06.3703402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhrytb2w/_remote_module_non_scriptable.py 2022-08-17T12:51:06.5555771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:06.6035530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:06.6222272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:06.6331800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:07.2660617Z ok (2.380s) 2022-08-17T12:51:07.2660894Z 2022-08-17T12:51:07.2661389Z ---------------------------------------------------------------------- 2022-08-17T12:51:07.2661662Z Ran 1 test in 2.380s 2022-08-17T12:51:07.2661787Z 2022-08-17T12:51:07.2661837Z OK 2022-08-17T12:51:07.2661935Z 2022-08-17T12:51:07.2662278Z Generating XML reports... 2022-08-17T12:51:07.2694373Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125104.xml 2022-08-17T12:51:08.5760349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:08.5760842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:08.5763218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:08.5763584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:08.7275832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcgkssdm 2022-08-17T12:51:08.7278938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcgkssdm/_remote_module_non_scriptable.py 2022-08-17T12:51:08.9922150Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:08.9933016Z 2022-08-17T12:51:08.9933119Z Running tests... 2022-08-17T12:51:08.9933888Z ---------------------------------------------------------------------- 2022-08-17T12:51:09.2582050Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10575 2022-08-17T12:51:09.2611094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10576 2022-08-17T12:51:09.2640384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10577 2022-08-17T12:51:09.2672324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10578 2022-08-17T12:51:10.3063110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:10.3063756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:10.3067258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:10.3067912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:10.3103699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:10.3104345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:10.3108315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:10.3108895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:10.3180582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:10.3181170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:10.3185706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:10.3186343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:10.3273345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:10.3274020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:10.3277292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:10.3277982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:10.4677373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj4n1j0u 2022-08-17T12:51:10.4678625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj4n1j0u/_remote_module_non_scriptable.py 2022-08-17T12:51:10.4755348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qet8qpj 2022-08-17T12:51:10.4756531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qet8qpj/_remote_module_non_scriptable.py 2022-08-17T12:51:10.4781932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10fmb8sc 2022-08-17T12:51:10.4784395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10fmb8sc/_remote_module_non_scriptable.py 2022-08-17T12:51:10.4963087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrnytho8 2022-08-17T12:51:10.4963946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrnytho8/_remote_module_non_scriptable.py 2022-08-17T12:51:10.7282325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:10.7353672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:10.7356947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:10.7543138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:11.3730069Z ok (2.379s) 2022-08-17T12:51:11.3730331Z 2022-08-17T12:51:11.3730798Z ---------------------------------------------------------------------- 2022-08-17T12:51:11.3731067Z Ran 1 test in 2.380s 2022-08-17T12:51:11.3731193Z 2022-08-17T12:51:11.3731257Z OK 2022-08-17T12:51:11.3731595Z 2022-08-17T12:51:11.3731773Z Generating XML reports... 2022-08-17T12:51:11.3764948Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125108.xml 2022-08-17T12:51:12.6842998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:12.6843430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:12.6846060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:12.6846448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:12.8376566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bxcha5w 2022-08-17T12:51:12.8379740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bxcha5w/_remote_module_non_scriptable.py 2022-08-17T12:51:13.1044534Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:13.1055788Z 2022-08-17T12:51:13.1055926Z Running tests... 2022-08-17T12:51:13.1056355Z ---------------------------------------------------------------------- 2022-08-17T12:51:13.3706960Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10798 2022-08-17T12:51:13.3735734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10799 2022-08-17T12:51:13.3764993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10800 2022-08-17T12:51:13.3798671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10801 2022-08-17T12:51:14.3776690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:14.3777410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:14.3780845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:14.3781506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:14.3829788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:14.3830401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:14.3835002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:14.3835647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:14.4310367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:14.4310977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:14.4313967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:14.4314490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:14.4404510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:14.4405046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:14.4408832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:14.4409497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:14.5443526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qn5uqv3 2022-08-17T12:51:14.5444382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qn5uqv3/_remote_module_non_scriptable.py 2022-08-17T12:51:14.5564354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sokcy1o 2022-08-17T12:51:14.5565115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sokcy1o/_remote_module_non_scriptable.py 2022-08-17T12:51:14.5915438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8r7shmk 2022-08-17T12:51:14.5917006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8r7shmk/_remote_module_non_scriptable.py 2022-08-17T12:51:14.5990346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhvflxpp 2022-08-17T12:51:14.5991454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhvflxpp/_remote_module_non_scriptable.py 2022-08-17T12:51:14.8062049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:14.8155131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:14.8549522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:14.8591755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:15.4856116Z ok (2.380s) 2022-08-17T12:51:15.4856351Z 2022-08-17T12:51:15.4856770Z ---------------------------------------------------------------------- 2022-08-17T12:51:15.4857020Z Ran 1 test in 2.380s 2022-08-17T12:51:15.4857147Z 2022-08-17T12:51:15.4857212Z OK 2022-08-17T12:51:15.4857309Z 2022-08-17T12:51:15.4857406Z Generating XML reports... 2022-08-17T12:51:15.4891536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125113.xml 2022-08-17T12:51:16.8168255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:16.8168694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:16.8171039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:16.8171429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:16.9716715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvbnnu4i 2022-08-17T12:51:16.9720727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvbnnu4i/_remote_module_non_scriptable.py 2022-08-17T12:51:17.2397615Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:17.2409637Z 2022-08-17T12:51:17.2410078Z Running tests... 2022-08-17T12:51:17.2410569Z ---------------------------------------------------------------------- 2022-08-17T12:51:17.5060738Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11021 2022-08-17T12:51:17.5091722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11022 2022-08-17T12:51:17.5122071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11023 2022-08-17T12:51:17.5154515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11024 2022-08-17T12:51:18.5466650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:18.5467341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:18.5469423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:18.5470064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:18.5484940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:18.5485545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:18.5490135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:18.5491340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:18.5692880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:18.5693550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:18.5696081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:18.5697138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:18.5815162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:18.5815766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:18.5830630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:18.5831473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:18.7058649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp340nobtd 2022-08-17T12:51:18.7059490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp340nobtd/_remote_module_non_scriptable.py 2022-08-17T12:51:18.7091708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2bglsbn 2022-08-17T12:51:18.7093026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2bglsbn/_remote_module_non_scriptable.py 2022-08-17T12:51:18.7327916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gnxy9xy 2022-08-17T12:51:18.7328748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gnxy9xy/_remote_module_non_scriptable.py 2022-08-17T12:51:18.7587134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rjbyepc 2022-08-17T12:51:18.7588456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rjbyepc/_remote_module_non_scriptable.py 2022-08-17T12:51:18.9642100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:18.9702438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:18.9942305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:19.0199905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:19.3208628Z ok (2.080s) 2022-08-17T12:51:19.3208861Z 2022-08-17T12:51:19.3209173Z ---------------------------------------------------------------------- 2022-08-17T12:51:19.3209638Z Ran 1 test in 2.080s 2022-08-17T12:51:19.3209760Z 2022-08-17T12:51:19.3209826Z OK 2022-08-17T12:51:19.3209925Z 2022-08-17T12:51:19.3210018Z Generating XML reports... 2022-08-17T12:51:19.3242775Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125117.xml 2022-08-17T12:51:20.6234417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:20.6235145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:20.6237846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:20.6238441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:20.7771993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o8fhhx2 2022-08-17T12:51:20.7775717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o8fhhx2/_remote_module_non_scriptable.py 2022-08-17T12:51:21.0403532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:21.0414748Z 2022-08-17T12:51:21.0415165Z Running tests... 2022-08-17T12:51:21.0416142Z ---------------------------------------------------------------------- 2022-08-17T12:51:21.3063846Z test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11076 2022-08-17T12:51:21.3093555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11077 2022-08-17T12:51:21.3122474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11078 2022-08-17T12:51:21.3155368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11079 2022-08-17T12:51:22.3530034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:22.3530772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:22.3533923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:22.3534608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:22.3784003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:22.3784710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:22.3786184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:22.3786814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:22.3855759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:22.3856466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:22.3859371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:22.3860045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:22.4172211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:22.4172895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:22.4174357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:22.4175009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:22.5119446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zfmfqml 2022-08-17T12:51:22.5120436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zfmfqml/_remote_module_non_scriptable.py 2022-08-17T12:51:22.5383105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_gxf1ts 2022-08-17T12:51:22.5384176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_gxf1ts/_remote_module_non_scriptable.py 2022-08-17T12:51:22.5450282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbcds1j8 2022-08-17T12:51:22.5451756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbcds1j8/_remote_module_non_scriptable.py 2022-08-17T12:51:22.5769658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20j3cg70 2022-08-17T12:51:22.5770429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20j3cg70/_remote_module_non_scriptable.py 2022-08-17T12:51:22.7731718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:22.7997003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:22.8066177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:22.8382829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:24.0840902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:51:24.0940057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:51:24.0941275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:51:24.0943051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:51:24.0944368Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:51:24.0945680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:51:24.0947844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:51:24.0949565Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:51:24.5231111Z ok (3.481s) 2022-08-17T12:51:24.5231351Z 2022-08-17T12:51:24.5231685Z ---------------------------------------------------------------------- 2022-08-17T12:51:24.5231942Z Ran 1 test in 3.482s 2022-08-17T12:51:24.5232139Z 2022-08-17T12:51:24.5232206Z OK 2022-08-17T12:51:24.5232300Z 2022-08-17T12:51:24.5232397Z Generating XML reports... 2022-08-17T12:51:24.5266118Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125121.xml 2022-08-17T12:51:25.8305703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:25.8306082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:25.8308425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:25.8308806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:25.9821934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckn1pqox 2022-08-17T12:51:25.9824578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckn1pqox/_remote_module_non_scriptable.py 2022-08-17T12:51:26.2478823Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:26.2489860Z 2022-08-17T12:51:26.2489940Z Running tests... 2022-08-17T12:51:26.2490526Z ---------------------------------------------------------------------- 2022-08-17T12:51:26.5103800Z test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11311 2022-08-17T12:51:26.5131918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11312 2022-08-17T12:51:26.5159845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11313 2022-08-17T12:51:26.5190532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11314 2022-08-17T12:51:27.5096281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:27.5096930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:27.5099763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:27.5100420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:27.5363621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:27.5364270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:27.5367492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:27.5368142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:27.5474474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:27.5475178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:27.5478991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:27.5479633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:27.5934245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:27.5934807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:27.5937777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:27.5938358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:27.6684471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25dam35y 2022-08-17T12:51:27.6685286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25dam35y/_remote_module_non_scriptable.py 2022-08-17T12:51:27.6948067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04xr1z4y 2022-08-17T12:51:27.6948902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04xr1z4y/_remote_module_non_scriptable.py 2022-08-17T12:51:27.7045687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr1p0a8q 2022-08-17T12:51:27.7046929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr1p0a8q/_remote_module_non_scriptable.py 2022-08-17T12:51:27.7509297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jq2yvk_ 2022-08-17T12:51:27.7510157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jq2yvk_/_remote_module_non_scriptable.py 2022-08-17T12:51:27.9306567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:27.9552062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:27.9627601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:28.0113931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:28.6250565Z ok (2.376s) 2022-08-17T12:51:28.6250754Z 2022-08-17T12:51:28.6251152Z ---------------------------------------------------------------------- 2022-08-17T12:51:28.6251427Z Ran 1 test in 2.376s 2022-08-17T12:51:28.6251552Z 2022-08-17T12:51:28.6251616Z OK 2022-08-17T12:51:28.6251715Z 2022-08-17T12:51:28.6251809Z Generating XML reports... 2022-08-17T12:51:28.6284218Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125126.xml 2022-08-17T12:51:29.9375139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:29.9375550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:29.9378306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:29.9378686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:30.0876712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5662y01 2022-08-17T12:51:30.0878949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5662y01/_remote_module_non_scriptable.py 2022-08-17T12:51:30.3487132Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:30.3498201Z 2022-08-17T12:51:30.3498288Z Running tests... 2022-08-17T12:51:30.3499262Z ---------------------------------------------------------------------- 2022-08-17T12:51:30.3509460Z test_default_timeout_used (__main__.TensorPipeRpcTest) 2022-08-17T12:51:30.6095064Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11538 2022-08-17T12:51:30.6122681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11539 2022-08-17T12:51:30.6151168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11540 2022-08-17T12:51:30.6183230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11541 2022-08-17T12:51:31.6627416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:31.6628011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:31.6628812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:31.6629373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:31.6630438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:31.6631022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:31.6631830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:31.6632419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:31.6706245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:31.6706927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:31.6710928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:31.6711592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:31.6715389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:31.6716011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:31.6720461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:31.6721104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:31.8251842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqjs63mj 2022-08-17T12:51:31.8252801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqjs63mj/_remote_module_non_scriptable.py 2022-08-17T12:51:31.8259790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1by9m02 2022-08-17T12:51:31.8261415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1by9m02/_remote_module_non_scriptable.py 2022-08-17T12:51:31.8315816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k_uu761 2022-08-17T12:51:31.8318238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k_uu761/_remote_module_non_scriptable.py 2022-08-17T12:51:31.8318970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemvez3__ 2022-08-17T12:51:31.8320865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemvez3__/_remote_module_non_scriptable.py 2022-08-17T12:51:32.0877167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:32.0900143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:32.0915465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:32.0937180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:34.7272470Z ok (4.377s) 2022-08-17T12:51:34.7272664Z 2022-08-17T12:51:34.7273189Z ---------------------------------------------------------------------- 2022-08-17T12:51:34.7273630Z Ran 1 test in 4.377s 2022-08-17T12:51:34.7274002Z 2022-08-17T12:51:34.7274070Z OK 2022-08-17T12:51:34.7274158Z 2022-08-17T12:51:34.7274260Z Generating XML reports... 2022-08-17T12:51:34.7307198Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125130.xml 2022-08-17T12:51:36.0246366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:36.0246751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:36.0249072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:36.0249468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:36.1780772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1p9g08c 2022-08-17T12:51:36.1784225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1p9g08c/_remote_module_non_scriptable.py 2022-08-17T12:51:36.4428742Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:36.4439200Z 2022-08-17T12:51:36.4439342Z Running tests... 2022-08-17T12:51:36.4439835Z ---------------------------------------------------------------------- 2022-08-17T12:51:36.7094316Z test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11761 2022-08-17T12:51:36.7121871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11762 2022-08-17T12:51:36.7149695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11763 2022-08-17T12:51:36.7180703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11764 2022-08-17T12:51:37.7037802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:37.7038448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:37.7040627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:37.7041219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:37.7444006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:37.7444679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:37.7447478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:37.7448608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:37.7833240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:37.7833882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:37.7835315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:37.7835931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:37.7978133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:37.7978775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:37.7981888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:37.7982546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:37.8594362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50_53cjf 2022-08-17T12:51:37.8595261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50_53cjf/_remote_module_non_scriptable.py 2022-08-17T12:51:37.9037632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmper4rrjb4 2022-08-17T12:51:37.9038418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmper4rrjb4/_remote_module_non_scriptable.py 2022-08-17T12:51:37.9401795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_jyxgbp 2022-08-17T12:51:37.9402676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_jyxgbp/_remote_module_non_scriptable.py 2022-08-17T12:51:37.9552847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7ns30uu 2022-08-17T12:51:37.9554225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7ns30uu/_remote_module_non_scriptable.py 2022-08-17T12:51:38.1162291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:38.1670787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:38.1982369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:38.2131508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:38.8237557Z ok (2.380s) 2022-08-17T12:51:38.8237740Z 2022-08-17T12:51:38.8238162Z ---------------------------------------------------------------------- 2022-08-17T12:51:38.8238435Z Ran 1 test in 2.380s 2022-08-17T12:51:38.8238542Z 2022-08-17T12:51:38.8238605Z OK 2022-08-17T12:51:38.8238706Z 2022-08-17T12:51:38.8238803Z Generating XML reports... 2022-08-17T12:51:38.8272366Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125136.xml 2022-08-17T12:51:40.1260546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:40.1260948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:40.1263177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:40.1263560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:40.2781880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jcepdd9 2022-08-17T12:51:40.2785386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jcepdd9/_remote_module_non_scriptable.py 2022-08-17T12:51:40.5418083Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:40.5428883Z 2022-08-17T12:51:40.5428974Z Running tests... 2022-08-17T12:51:40.5429736Z ---------------------------------------------------------------------- 2022-08-17T12:51:40.8080629Z test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11984 2022-08-17T12:51:40.8110384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11985 2022-08-17T12:51:40.8139368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11986 2022-08-17T12:51:40.8171218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11987 2022-08-17T12:51:41.9392384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:41.9393050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:41.9396229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:41.9396835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:41.9414266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:41.9414873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:41.9419917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:41.9420605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:41.9445067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:41.9445712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:41.9449636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:41.9450297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:41.9713169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:41.9713904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:41.9716998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:41.9717654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:42.0989788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfc89q19 2022-08-17T12:51:42.0991067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfc89q19/_remote_module_non_scriptable.py 2022-08-17T12:51:42.1023667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncf_3igb 2022-08-17T12:51:42.1025741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncf_3igb/_remote_module_non_scriptable.py 2022-08-17T12:51:42.1033942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4y3qsy8 2022-08-17T12:51:42.1036132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4y3qsy8/_remote_module_non_scriptable.py 2022-08-17T12:51:42.1316820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fmnbegc 2022-08-17T12:51:42.1317683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fmnbegc/_remote_module_non_scriptable.py 2022-08-17T12:51:42.3627438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:42.3640782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:42.3649753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:42.3972244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:43.0231237Z ok (2.480s) 2022-08-17T12:51:43.0231487Z 2022-08-17T12:51:43.0232024Z ---------------------------------------------------------------------- 2022-08-17T12:51:43.0232548Z Ran 1 test in 2.480s 2022-08-17T12:51:43.0232669Z 2022-08-17T12:51:43.0232735Z OK 2022-08-17T12:51:43.0232831Z 2022-08-17T12:51:43.0232929Z Generating XML reports... 2022-08-17T12:51:43.0267715Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125140.xml 2022-08-17T12:51:44.3198889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:44.3199283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:44.3202117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:44.3202504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:44.4701577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_rw297q 2022-08-17T12:51:44.4704420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_rw297q/_remote_module_non_scriptable.py 2022-08-17T12:51:44.7308368Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:44.7319406Z 2022-08-17T12:51:44.7319543Z Running tests... 2022-08-17T12:51:44.7320473Z ---------------------------------------------------------------------- 2022-08-17T12:51:44.9939379Z test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12207 2022-08-17T12:51:44.9967997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12208 2022-08-17T12:51:44.9996382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12209 2022-08-17T12:51:45.0027523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12210 2022-08-17T12:51:46.0423339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:46.0424002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:46.0425831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:46.0426489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:46.0471568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:46.0472344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:46.0475775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:46.0476449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:46.0482112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:46.0482785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:46.0487514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:46.0488149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:46.0634189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:46.0637886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:46.0638743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:46.0639375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:46.2012149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf50dy9v 2022-08-17T12:51:46.2013013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf50dy9v/_remote_module_non_scriptable.py 2022-08-17T12:51:46.2058623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp148u1i5z 2022-08-17T12:51:46.2060073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp148u1i5z/_remote_module_non_scriptable.py 2022-08-17T12:51:46.2083435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqcv2ul3 2022-08-17T12:51:46.2086199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqcv2ul3/_remote_module_non_scriptable.py 2022-08-17T12:51:46.2222825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudnh4bec 2022-08-17T12:51:46.2223713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudnh4bec/_remote_module_non_scriptable.py 2022-08-17T12:51:46.4641862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:46.4709975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:46.4758450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:46.4897668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:46.9081969Z ok (2.176s) 2022-08-17T12:51:46.9082235Z 2022-08-17T12:51:46.9082825Z ---------------------------------------------------------------------- 2022-08-17T12:51:46.9083084Z Ran 1 test in 2.176s 2022-08-17T12:51:46.9083207Z 2022-08-17T12:51:46.9083270Z OK 2022-08-17T12:51:46.9083367Z 2022-08-17T12:51:46.9083461Z Generating XML reports... 2022-08-17T12:51:46.9117562Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125144.xml 2022-08-17T12:51:48.2116028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:48.2116708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:48.2119622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:48.2120209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:48.3630213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1dpj8y4 2022-08-17T12:51:48.3633810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1dpj8y4/_remote_module_non_scriptable.py 2022-08-17T12:51:48.6255947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:48.6267497Z 2022-08-17T12:51:48.6267961Z Running tests... 2022-08-17T12:51:48.6268380Z ---------------------------------------------------------------------- 2022-08-17T12:51:48.8916424Z test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12342 2022-08-17T12:51:48.8944319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12343 2022-08-17T12:51:48.8973514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12344 2022-08-17T12:51:48.9005045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12345 2022-08-17T12:51:49.9672338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:49.9672997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:49.9675892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:49.9676585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:49.9787329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:49.9787980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:49.9791564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:49.9792240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:49.9870984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:49.9871614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:49.9875877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:49.9876539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:50.0378892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:50.0379567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:50.0382949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:50.0383522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:50.1319615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1hnr_li 2022-08-17T12:51:50.1320597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1hnr_li/_remote_module_non_scriptable.py 2022-08-17T12:51:50.1372688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_s17twq 2022-08-17T12:51:50.1375023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_s17twq/_remote_module_non_scriptable.py 2022-08-17T12:51:50.1630934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0phqq7yx 2022-08-17T12:51:50.1631901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0phqq7yx/_remote_module_non_scriptable.py 2022-08-17T12:51:50.2000182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw1_cggb 2022-08-17T12:51:50.2000939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw1_cggb/_remote_module_non_scriptable.py 2022-08-17T12:51:50.3926024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:50.3962116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:50.4195664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:50.4600535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:50.9063051Z ok (2.279s) 2022-08-17T12:51:50.9063316Z 2022-08-17T12:51:50.9063774Z ---------------------------------------------------------------------- 2022-08-17T12:51:50.9064057Z Ran 1 test in 2.279s 2022-08-17T12:51:50.9064179Z 2022-08-17T12:51:50.9064245Z OK 2022-08-17T12:51:50.9064344Z 2022-08-17T12:51:50.9064443Z Generating XML reports... 2022-08-17T12:51:50.9097105Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125148.xml 2022-08-17T12:51:52.1851716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:52.1852127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:52.1854631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:52.1855099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:52.3341611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu17e3kzz 2022-08-17T12:51:52.3345553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu17e3kzz/_remote_module_non_scriptable.py 2022-08-17T12:51:52.5993836Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:52.6004561Z 2022-08-17T12:51:52.6004662Z Running tests... 2022-08-17T12:51:52.6005769Z ---------------------------------------------------------------------- 2022-08-17T12:51:52.8615629Z test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12477 2022-08-17T12:51:52.8644404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12478 2022-08-17T12:51:52.8673395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12479 2022-08-17T12:51:52.8704570Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12480 2022-08-17T12:51:53.9106414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:53.9107014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:53.9110062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:53.9110743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:53.9236381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:53.9237009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:53.9240563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:53.9241262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:53.9286476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:53.9287114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:53.9290829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:53.9291493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:53.9558654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:53.9559343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:53.9562141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:53.9562774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:54.0709696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5j79lm_ 2022-08-17T12:51:54.0710737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5j79lm_/_remote_module_non_scriptable.py 2022-08-17T12:51:54.0854264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip1v2egn 2022-08-17T12:51:54.0855353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip1v2egn/_remote_module_non_scriptable.py 2022-08-17T12:51:54.0863648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujjjdm83 2022-08-17T12:51:54.0866102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujjjdm83/_remote_module_non_scriptable.py 2022-08-17T12:51:54.1135786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zrkrqzo 2022-08-17T12:51:54.1136880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zrkrqzo/_remote_module_non_scriptable.py 2022-08-17T12:51:54.3323527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:54.3466131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:54.3480735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:54.3987365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:55.0765503Z ok (2.476s) 2022-08-17T12:51:55.0765763Z 2022-08-17T12:51:55.0766464Z ---------------------------------------------------------------------- 2022-08-17T12:51:55.0766766Z Ran 1 test in 2.476s 2022-08-17T12:51:55.0766889Z 2022-08-17T12:51:55.0766954Z OK 2022-08-17T12:51:55.0767050Z 2022-08-17T12:51:55.0767146Z Generating XML reports... 2022-08-17T12:51:55.0801440Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125152.xml 2022-08-17T12:51:56.4012580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:56.4012978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:56.4015102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:56.4015659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:56.5536052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkmmsy22 2022-08-17T12:51:56.5539621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkmmsy22/_remote_module_non_scriptable.py 2022-08-17T12:51:56.8195639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:51:56.8206686Z 2022-08-17T12:51:56.8206871Z Running tests... 2022-08-17T12:51:56.8207489Z ---------------------------------------------------------------------- 2022-08-17T12:51:57.0857078Z test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12700 2022-08-17T12:51:57.0886345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12701 2022-08-17T12:51:57.0915005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12702 2022-08-17T12:51:57.0950035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12703 2022-08-17T12:51:58.1304087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:58.1304776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:58.1307599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:58.1308242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:58.1369014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:58.1369677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:58.1373439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:58.1374146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:58.1519224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:58.1519958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:58.1523716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:58.1524382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:58.1620092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:51:58.1620761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:51:58.1623735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:51:58.1624401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:51:58.2889302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbmbe0a0 2022-08-17T12:51:58.2890255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbmbe0a0/_remote_module_non_scriptable.py 2022-08-17T12:51:58.2969131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo43kpxe5 2022-08-17T12:51:58.2971090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo43kpxe5/_remote_module_non_scriptable.py 2022-08-17T12:51:58.3109322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8fza_5j 2022-08-17T12:51:58.3110076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8fza_5j/_remote_module_non_scriptable.py 2022-08-17T12:51:58.3198352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplioyy7vw 2022-08-17T12:51:58.3200035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplioyy7vw/_remote_module_non_scriptable.py 2022-08-17T12:51:58.5586989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:51:58.5707063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:51:58.5749382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:51:58.5825915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:51:58.8173640Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:51:58.8175809Z AttributeError("Can't get attribute 'foo_add' on Default RPC pickler does not serialize\n function code. Ensure that UDFs are defined on both caller and\n callee modules.") 2022-08-17T12:51:58.8177074Z Traceback (most recent call last): 2022-08-17T12:51:58.8178030Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 159, in deserialize 2022-08-17T12:51:58.8178699Z ret = unpickler.load() 2022-08-17T12:51:58.8180002Z AttributeError: Can't get attribute 'foo_add' on 2022-08-17T12:51:58.8180745Z 2022-08-17T12:51:58.8181072Z The above exception was the direct cause of the following exception: 2022-08-17T12:51:58.8181459Z 2022-08-17T12:51:58.8181678Z Traceback (most recent call last): 2022-08-17T12:51:58.8182564Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 205, in _run_function 2022-08-17T12:51:58.8183192Z raise python_udf 2022-08-17T12:51:58.8184625Z AttributeError: Can't get attribute 'foo_add' on Default RPC pickler does not serialize 2022-08-17T12:51:58.8185739Z function code. Ensure that UDFs are defined on both caller and 2022-08-17T12:51:58.8186250Z callee modules. 2022-08-17T12:51:58.8186502Z 2022-08-17T12:51:59.2007519Z ok (2.380s) 2022-08-17T12:51:59.2007676Z 2022-08-17T12:51:59.2008044Z ---------------------------------------------------------------------- 2022-08-17T12:51:59.2008325Z Ran 1 test in 2.380s 2022-08-17T12:51:59.2008461Z 2022-08-17T12:51:59.2008575Z OK 2022-08-17T12:51:59.2008704Z 2022-08-17T12:51:59.2008787Z Generating XML reports... 2022-08-17T12:51:59.2041570Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125156.xml 2022-08-17T12:52:00.4959144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:00.4959550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:00.4961320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:00.4962022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:00.6452054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0ufadmo 2022-08-17T12:52:00.6455694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0ufadmo/_remote_module_non_scriptable.py 2022-08-17T12:52:00.9067564Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:00.9079086Z 2022-08-17T12:52:00.9079333Z Running tests... 2022-08-17T12:52:00.9080009Z ---------------------------------------------------------------------- 2022-08-17T12:52:01.1661269Z test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12923 2022-08-17T12:52:01.1689015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12924 2022-08-17T12:52:01.1717604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12925 2022-08-17T12:52:01.1748302Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12926 2022-08-17T12:52:02.2188962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:02.2189926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:02.2193092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:02.2193754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:02.2205361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:02.2206089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:02.2210050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:02.2210711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:02.2324598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:02.2325149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:02.2328829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:02.2329454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:02.2423178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:02.2423843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:02.2427089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:02.2427737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:02.3789428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw_h7u8v 2022-08-17T12:52:02.3790462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw_h7u8v/_remote_module_non_scriptable.py 2022-08-17T12:52:02.3809115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o8dq66q 2022-08-17T12:52:02.3811670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o8dq66q/_remote_module_non_scriptable.py 2022-08-17T12:52:02.3931618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbml19mqi 2022-08-17T12:52:02.3933523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbml19mqi/_remote_module_non_scriptable.py 2022-08-17T12:52:02.4007240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_s_agxv 2022-08-17T12:52:02.4008427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_s_agxv/_remote_module_non_scriptable.py 2022-08-17T12:52:02.6417274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:02.6463644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:02.6654737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:02.6821590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:03.2806575Z ok (2.372s) 2022-08-17T12:52:03.2806746Z 2022-08-17T12:52:03.2807249Z ---------------------------------------------------------------------- 2022-08-17T12:52:03.2807734Z Ran 1 test in 2.373s 2022-08-17T12:52:03.2807856Z 2022-08-17T12:52:03.2807924Z OK 2022-08-17T12:52:03.2808020Z 2022-08-17T12:52:03.2808119Z Generating XML reports... 2022-08-17T12:52:03.2840855Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125200.xml 2022-08-17T12:52:04.5671087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:04.5671469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:04.5673207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:04.5673588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:04.7182630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1teamrq1 2022-08-17T12:52:04.7185840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1teamrq1/_remote_module_non_scriptable.py 2022-08-17T12:52:04.9823295Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:04.9833980Z 2022-08-17T12:52:04.9834135Z Running tests... 2022-08-17T12:52:04.9834486Z ---------------------------------------------------------------------- 2022-08-17T12:52:05.2459031Z test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13146 2022-08-17T12:52:05.2487623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13147 2022-08-17T12:52:05.2516257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13148 2022-08-17T12:52:05.2548030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13149 2022-08-17T12:52:06.3183165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:06.3183847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:06.3239088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:06.3239742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:06.3409391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:06.3410077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:06.3412880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:06.3413571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:06.3756947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:06.3757525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:06.3759099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:06.3759621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:06.3760319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:06.3761132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:06.3762333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:06.3762928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:06.4807106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr8kky6f 2022-08-17T12:52:06.4807978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr8kky6f/_remote_module_non_scriptable.py 2022-08-17T12:52:06.5012149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4duy8g1 2022-08-17T12:52:06.5013162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4duy8g1/_remote_module_non_scriptable.py 2022-08-17T12:52:06.5376248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u_im1m9 2022-08-17T12:52:06.5378879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kuipifn 2022-08-17T12:52:06.5379558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u_im1m9/_remote_module_non_scriptable.py 2022-08-17T12:52:06.5380519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kuipifn/_remote_module_non_scriptable.py 2022-08-17T12:52:06.7452478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:06.7813267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:06.8012542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:06.8020428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:07.0119393Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:52:07.0120231Z ValueError('Expected error') 2022-08-17T12:52:07.0120742Z Traceback (most recent call last): 2022-08-17T12:52:07.0121716Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:52:07.0122471Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:52:07.0123529Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:52:07.0124238Z raise ValueError(expected_err) 2022-08-17T12:52:07.0124694Z ValueError: Expected error 2022-08-17T12:52:07.0124968Z 2022-08-17T12:52:07.0315038Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:52:07.0315606Z ValueError('Expected error') 2022-08-17T12:52:07.0315993Z Traceback (most recent call last): 2022-08-17T12:52:07.0316705Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:52:07.0317290Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:52:07.0318093Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:52:07.0318621Z raise ValueError(expected_err) 2022-08-17T12:52:07.0318989Z ValueError: Expected error 2022-08-17T12:52:07.0319209Z 2022-08-17T12:52:07.0404379Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:52:07.0405139Z ValueError('Expected error') 2022-08-17T12:52:07.0405634Z Traceback (most recent call last): 2022-08-17T12:52:07.0406810Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:52:07.0407595Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:52:07.0408640Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:52:07.0409348Z raise ValueError(expected_err) 2022-08-17T12:52:07.0409836Z ValueError: Expected error 2022-08-17T12:52:07.0410483Z 2022-08-17T12:52:07.0438330Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:52:07.0439109Z ValueError('Expected error') 2022-08-17T12:52:07.0439781Z Traceback (most recent call last): 2022-08-17T12:52:07.0440897Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:52:07.0441841Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:52:07.0443070Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:52:07.0443958Z raise ValueError(expected_err) 2022-08-17T12:52:07.0444420Z ValueError: Expected error 2022-08-17T12:52:07.0444708Z 2022-08-17T12:52:07.4609272Z ok (2.477s) 2022-08-17T12:52:07.4609514Z 2022-08-17T12:52:07.4609856Z ---------------------------------------------------------------------- 2022-08-17T12:52:07.4610115Z Ran 1 test in 2.477s 2022-08-17T12:52:07.4610292Z 2022-08-17T12:52:07.4610361Z OK 2022-08-17T12:52:07.4610475Z 2022-08-17T12:52:07.4610571Z Generating XML reports... 2022-08-17T12:52:07.4643835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125204.xml 2022-08-17T12:52:08.7769268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:08.7769741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:08.7772105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:08.7772513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:08.9327485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphx_d8hi 2022-08-17T12:52:08.9329882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphx_d8hi/_remote_module_non_scriptable.py 2022-08-17T12:52:09.1961050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:09.1971201Z 2022-08-17T12:52:09.1971340Z Running tests... 2022-08-17T12:52:09.1971970Z ---------------------------------------------------------------------- 2022-08-17T12:52:09.4594760Z test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13369 2022-08-17T12:52:09.4622817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13370 2022-08-17T12:52:09.4651394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13371 2022-08-17T12:52:09.4683655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13372 2022-08-17T12:52:10.5223604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:10.5224246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:10.5227958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:10.5228606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:10.5229395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:10.5230004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:10.5231595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:10.5232238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:10.5444354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:10.5447860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:10.5448745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:10.5449632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:10.5458920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:10.5459547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:10.5463664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:10.5464325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:10.6808507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1pxu0ud 2022-08-17T12:52:10.6809867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1pxu0ud/_remote_module_non_scriptable.py 2022-08-17T12:52:10.6821137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2er45odk 2022-08-17T12:52:10.6822747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2er45odk/_remote_module_non_scriptable.py 2022-08-17T12:52:10.7046525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgohigfeg 2022-08-17T12:52:10.7047627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgohigfeg/_remote_module_non_scriptable.py 2022-08-17T12:52:10.7051662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohx3olg0 2022-08-17T12:52:10.7053973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohx3olg0/_remote_module_non_scriptable.py 2022-08-17T12:52:10.9418171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:10.9449124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:10.9688198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:10.9900156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:11.5740073Z ok (2.377s) 2022-08-17T12:52:11.5740522Z 2022-08-17T12:52:11.5741524Z ---------------------------------------------------------------------- 2022-08-17T12:52:11.5741978Z Ran 1 test in 2.377s 2022-08-17T12:52:11.5742104Z 2022-08-17T12:52:11.5742188Z OK 2022-08-17T12:52:11.5742291Z 2022-08-17T12:52:11.5742385Z Generating XML reports... 2022-08-17T12:52:11.5776898Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125209.xml 2022-08-17T12:52:12.8793915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:12.8794309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:12.8796857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:12.8797253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:13.0349620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeab2d5wl 2022-08-17T12:52:13.0352407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeab2d5wl/_remote_module_non_scriptable.py 2022-08-17T12:52:13.3006827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:13.3017397Z 2022-08-17T12:52:13.3017556Z Running tests... 2022-08-17T12:52:13.3018172Z ---------------------------------------------------------------------- 2022-08-17T12:52:13.5667110Z test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13592 2022-08-17T12:52:13.5696569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13593 2022-08-17T12:52:13.5725512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13594 2022-08-17T12:52:13.5757031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13595 2022-08-17T12:52:14.6145192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:14.6145892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:14.6148986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:14.6149641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:14.6576522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:14.6577179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:14.6580244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:14.6580903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:14.6907127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:14.6907888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:14.6909906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:14.6910631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:14.7068867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:14.7069534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:14.7072468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:14.7073019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:14.7761303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqi31ev3 2022-08-17T12:52:14.7762515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqi31ev3/_remote_module_non_scriptable.py 2022-08-17T12:52:14.8202596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiq2h25t 2022-08-17T12:52:14.8203323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiq2h25t/_remote_module_non_scriptable.py 2022-08-17T12:52:14.8540552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mrbbju3 2022-08-17T12:52:14.8541312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mrbbju3/_remote_module_non_scriptable.py 2022-08-17T12:52:14.8690370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlx4_cbz 2022-08-17T12:52:14.8691171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlx4_cbz/_remote_module_non_scriptable.py 2022-08-17T12:52:15.0424341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:15.0839969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:15.1199966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:15.1355251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:15.7817141Z ok (2.480s) 2022-08-17T12:52:15.7817368Z 2022-08-17T12:52:15.7817899Z ---------------------------------------------------------------------- 2022-08-17T12:52:15.7818369Z Ran 1 test in 2.480s 2022-08-17T12:52:15.7818502Z 2022-08-17T12:52:15.7818553Z OK 2022-08-17T12:52:15.7818648Z 2022-08-17T12:52:15.7818747Z Generating XML reports... 2022-08-17T12:52:15.7851483Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125213.xml 2022-08-17T12:52:17.1007587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:17.1008316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:17.1010414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:17.1010952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:17.2556131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp816gdlst 2022-08-17T12:52:17.2558639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp816gdlst/_remote_module_non_scriptable.py 2022-08-17T12:52:17.5213953Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:17.5225059Z 2022-08-17T12:52:17.5225222Z Running tests... 2022-08-17T12:52:17.5225666Z ---------------------------------------------------------------------- 2022-08-17T12:52:17.7810573Z test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s) 2022-08-17T12:52:17.7811071Z 2022-08-17T12:52:17.7811304Z ---------------------------------------------------------------------- 2022-08-17T12:52:17.7811569Z Ran 1 test in 0.258s 2022-08-17T12:52:17.7811690Z 2022-08-17T12:52:17.7811767Z OK (skipped=1) 2022-08-17T12:52:17.7811868Z 2022-08-17T12:52:17.7811958Z Generating XML reports... 2022-08-17T12:52:17.7836247Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125217.xml 2022-08-17T12:52:19.0351558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:19.0351981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:19.0354615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:19.0355061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:19.1966054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08l6r1nt 2022-08-17T12:52:19.1970258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08l6r1nt/_remote_module_non_scriptable.py 2022-08-17T12:52:19.4663908Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:19.4674232Z 2022-08-17T12:52:19.4674475Z Running tests... 2022-08-17T12:52:19.4674949Z ---------------------------------------------------------------------- 2022-08-17T12:52:19.7340257Z test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13825 2022-08-17T12:52:19.7368786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13826 2022-08-17T12:52:19.7396149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13827 2022-08-17T12:52:19.7427694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13828 2022-08-17T12:52:20.7382576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:20.7383219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:20.7386498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:20.7387161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:20.7387975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:20.7388871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:20.7391600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:20.7392229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:20.7456585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:20.7457235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:20.7460437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:20.7461126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:20.7904362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:20.7904991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:20.7908448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:20.7908976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:20.8998788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pton47h 2022-08-17T12:52:20.9000831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pton47h/_remote_module_non_scriptable.py 2022-08-17T12:52:20.9008490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpcccy_g 2022-08-17T12:52:20.9010485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpcccy_g/_remote_module_non_scriptable.py 2022-08-17T12:52:20.9053507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72wy5vv0 2022-08-17T12:52:20.9055314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72wy5vv0/_remote_module_non_scriptable.py 2022-08-17T12:52:20.9499086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fa57ga9 2022-08-17T12:52:20.9499862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fa57ga9/_remote_module_non_scriptable.py 2022-08-17T12:52:21.1653429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:21.1667829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:21.1668577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:21.2144640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:21.8485198Z ok (2.381s) 2022-08-17T12:52:21.8485421Z 2022-08-17T12:52:21.8486087Z ---------------------------------------------------------------------- 2022-08-17T12:52:21.8486534Z Ran 1 test in 2.381s 2022-08-17T12:52:21.8486705Z 2022-08-17T12:52:21.8486811Z OK 2022-08-17T12:52:21.8486994Z 2022-08-17T12:52:21.8487142Z Generating XML reports... 2022-08-17T12:52:21.8521556Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125219.xml 2022-08-17T12:52:23.1678285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:23.1678688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:23.1681198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:23.1681563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:23.3210001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6aocc95f 2022-08-17T12:52:23.3213063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6aocc95f/_remote_module_non_scriptable.py 2022-08-17T12:52:23.5856336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:23.5866894Z 2022-08-17T12:52:23.5867183Z Running tests... 2022-08-17T12:52:23.5867800Z ---------------------------------------------------------------------- 2022-08-17T12:52:23.5870415Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest) 2022-08-17T12:52:23.8500948Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14048 2022-08-17T12:52:23.8530338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14049 2022-08-17T12:52:23.8557730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14050 2022-08-17T12:52:23.8589526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14051 2022-08-17T12:52:24.9179715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:24.9180345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:24.9183135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:24.9183735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:24.9761169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:24.9761775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:24.9765412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:24.9766276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:24.9783634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:24.9784300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:24.9788041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:24.9788661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:24.9917682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:24.9918376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:24.9920497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:24.9921139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:25.0769511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ehb27mw 2022-08-17T12:52:25.0770590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ehb27mw/_remote_module_non_scriptable.py 2022-08-17T12:52:25.1352730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyuoqnjo 2022-08-17T12:52:25.1353479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyuoqnjo/_remote_module_non_scriptable.py 2022-08-17T12:52:25.1386184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_9o9m2j 2022-08-17T12:52:25.1388277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_9o9m2j/_remote_module_non_scriptable.py 2022-08-17T12:52:25.1507233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv9je1zf 2022-08-17T12:52:25.1508301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv9je1zf/_remote_module_non_scriptable.py 2022-08-17T12:52:25.3375863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:25.3939668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:25.3982757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:25.4089153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:26.1648675Z ok (2.578s) 2022-08-17T12:52:26.1648906Z 2022-08-17T12:52:26.1649421Z ---------------------------------------------------------------------- 2022-08-17T12:52:26.1649905Z Ran 1 test in 2.578s 2022-08-17T12:52:26.1650030Z 2022-08-17T12:52:26.1650094Z OK 2022-08-17T12:52:26.1650193Z 2022-08-17T12:52:26.1650288Z Generating XML reports... 2022-08-17T12:52:26.1684010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125223.xml 2022-08-17T12:52:27.4372007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:27.4372409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:27.4375070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:27.4375489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:27.5983171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkop8_ss 2022-08-17T12:52:27.5986404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkop8_ss/_remote_module_non_scriptable.py 2022-08-17T12:52:27.8611996Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:27.8622419Z 2022-08-17T12:52:27.8622525Z Running tests... 2022-08-17T12:52:27.8623133Z ---------------------------------------------------------------------- 2022-08-17T12:52:28.1255352Z test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14271 2022-08-17T12:52:28.1283716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14272 2022-08-17T12:52:28.1313492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14273 2022-08-17T12:52:28.1346381Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14274 2022-08-17T12:52:29.1528554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:29.1529632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:29.1541001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:29.1544514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:29.1898687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:29.1899329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:29.1901982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:29.1902657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:29.1931244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:29.1931883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:29.1936094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:29.1936819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:29.2035035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:29.2035584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:29.2038806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:29.2039637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:29.3136895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe75iedj 2022-08-17T12:52:29.3138150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe75iedj/_remote_module_non_scriptable.py 2022-08-17T12:52:29.3494138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjul3h7_ 2022-08-17T12:52:29.3495107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjul3h7_/_remote_module_non_scriptable.py 2022-08-17T12:52:29.3532207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbfudh0m 2022-08-17T12:52:29.3533813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbfudh0m/_remote_module_non_scriptable.py 2022-08-17T12:52:29.3643055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgga_u8kd 2022-08-17T12:52:29.3644645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgga_u8kd/_remote_module_non_scriptable.py 2022-08-17T12:52:29.5775439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:29.6066751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:29.6146655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:29.6260365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:29.8960150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:52:29.9163609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:52:29.9264248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:52:29.9265229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:52:29.9266613Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:29.9267865Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:29.9269107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:29.9270347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:29.9375631Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:52:29.9379931Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-08-17T12:52:29.9380994Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:52:29.9382046Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:52:29.9382902Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:52:29.9383598Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:52:29.9384273Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:52:29.9395062Z [W tensorpipe_agent.cpp:916] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62) 2022-08-17T12:52:30.3405184Z ok (2.478s) 2022-08-17T12:52:30.3405473Z 2022-08-17T12:52:30.3406008Z ---------------------------------------------------------------------- 2022-08-17T12:52:30.3406281Z Ran 1 test in 2.478s 2022-08-17T12:52:30.3406405Z 2022-08-17T12:52:30.3406457Z OK 2022-08-17T12:52:30.3406558Z 2022-08-17T12:52:30.3406652Z Generating XML reports... 2022-08-17T12:52:30.3442096Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125227.xml 2022-08-17T12:52:31.6315816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:31.6316225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:31.6318754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:31.6319376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:31.7869329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2g35j68 2022-08-17T12:52:31.7872864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2g35j68/_remote_module_non_scriptable.py 2022-08-17T12:52:32.0534092Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:32.0544843Z 2022-08-17T12:52:32.0544987Z Running tests... 2022-08-17T12:52:32.0545630Z ---------------------------------------------------------------------- 2022-08-17T12:52:32.3180735Z test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14506 2022-08-17T12:52:32.3208540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14507 2022-08-17T12:52:32.3239242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14508 2022-08-17T12:52:32.3270997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14509 2022-08-17T12:52:33.3795952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:33.3796612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:33.3800066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:33.3800764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:33.3854593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:33.3855273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:33.3859210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:33.3859899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:33.3885288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:33.3886199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:33.3889228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:33.3889892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:33.3988952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:33.3989896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:33.3992783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:33.3993415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:33.5387167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp790z4zxw 2022-08-17T12:52:33.5387952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp790z4zxw/_remote_module_non_scriptable.py 2022-08-17T12:52:33.5450594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlbdxjhc 2022-08-17T12:52:33.5452160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlbdxjhc/_remote_module_non_scriptable.py 2022-08-17T12:52:33.5461087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfanxynx 2022-08-17T12:52:33.5463056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfanxynx/_remote_module_non_scriptable.py 2022-08-17T12:52:33.5564836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp61olh_4 2022-08-17T12:52:33.5566376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp61olh_4/_remote_module_non_scriptable.py 2022-08-17T12:52:33.7971032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:33.8015169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:33.8156853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:33.8231229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:34.0556973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:52:34.0559400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:52:34.0660895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:52:34.0662170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:34.0662862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:52:34.0663391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:34.0663945Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:34.0664486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:34.1098884Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:52:34.1101944Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-08-17T12:52:34.1102287Z 2022-08-17T12:52:34.1103267Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:52:34.1104867Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-08-17T12:52:34.1105216Z 2022-08-17T12:52:34.1106179Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:52:34.1107338Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-08-17T12:52:34.1107680Z 2022-08-17T12:52:34.1172396Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:52:34.1173962Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-08-17T12:52:34.1174386Z 2022-08-17T12:52:34.5331460Z ok (2.478s) 2022-08-17T12:52:34.5331683Z 2022-08-17T12:52:34.5332226Z ---------------------------------------------------------------------- 2022-08-17T12:52:34.5332653Z Ran 1 test in 2.479s 2022-08-17T12:52:34.5332783Z 2022-08-17T12:52:34.5332846Z OK 2022-08-17T12:52:34.5332942Z 2022-08-17T12:52:34.5333026Z Generating XML reports... 2022-08-17T12:52:34.5368784Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125232.xml 2022-08-17T12:52:35.8442559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:35.8442966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:35.8445688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:35.8446368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:35.9992399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryusrh44 2022-08-17T12:52:35.9995414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryusrh44/_remote_module_non_scriptable.py 2022-08-17T12:52:36.2619013Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:36.2630684Z 2022-08-17T12:52:36.2630975Z Running tests... 2022-08-17T12:52:36.2631421Z ---------------------------------------------------------------------- 2022-08-17T12:52:36.5235401Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14741 2022-08-17T12:52:36.5263018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14742 2022-08-17T12:52:36.5292564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14743 2022-08-17T12:52:36.5323630Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14744 2022-08-17T12:52:37.5542389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:37.5543035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:37.5546487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:37.5547164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:37.5591192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:37.5591839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:37.5596351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:37.5596974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:37.5655913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:37.5656548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:37.5659682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:37.5660382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:37.5889723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:37.5890433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:37.5894271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:37.5894966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:37.7141982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_yfzz8g 2022-08-17T12:52:37.7142684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_yfzz8g/_remote_module_non_scriptable.py 2022-08-17T12:52:37.7165345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a0n_ps9 2022-08-17T12:52:37.7167523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a0n_ps9/_remote_module_non_scriptable.py 2022-08-17T12:52:37.7230977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ppnzdb0 2022-08-17T12:52:37.7232852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ppnzdb0/_remote_module_non_scriptable.py 2022-08-17T12:52:37.7462171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcoz2qbsu 2022-08-17T12:52:37.7463221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcoz2qbsu/_remote_module_non_scriptable.py 2022-08-17T12:52:37.9739768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:37.9840825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:37.9864825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:38.0060993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:38.0315192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:52:38.0446999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:52:38.0447603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:52:38.0448171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:52:38.0449025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:38.0449591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:38.0450137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:38.0518116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:38.7384204Z ok (2.475s) 2022-08-17T12:52:38.7384380Z 2022-08-17T12:52:38.7384951Z ---------------------------------------------------------------------- 2022-08-17T12:52:38.7385425Z Ran 1 test in 2.475s 2022-08-17T12:52:38.7385594Z 2022-08-17T12:52:38.7387225Z OK 2022-08-17T12:52:38.7387419Z 2022-08-17T12:52:38.7387605Z Generating XML reports... 2022-08-17T12:52:38.7420409Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125236.xml 2022-08-17T12:52:40.0335431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:40.0335816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:40.0338580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:40.0338983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:40.1866367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtr6zwyw 2022-08-17T12:52:40.1869787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtr6zwyw/_remote_module_non_scriptable.py 2022-08-17T12:52:40.4498822Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:40.4509385Z 2022-08-17T12:52:40.4509471Z Running tests... 2022-08-17T12:52:40.4509964Z ---------------------------------------------------------------------- 2022-08-17T12:52:40.7138711Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14976 2022-08-17T12:52:40.7167097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14977 2022-08-17T12:52:40.7196156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14978 2022-08-17T12:52:40.7230252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14979 2022-08-17T12:52:41.7711851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:41.7712470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:41.7715221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:41.7715863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:41.8438140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:41.8438771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:41.8441663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:41.8442273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:41.8773835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:41.8774499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:41.8777221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:41.8777917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:41.8779163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:41.8779780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:41.8784473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:41.8785251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:41.9307943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryu579qt 2022-08-17T12:52:41.9309319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryu579qt/_remote_module_non_scriptable.py 2022-08-17T12:52:41.9999950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacmd2csg 2022-08-17T12:52:42.0000845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacmd2csg/_remote_module_non_scriptable.py 2022-08-17T12:52:42.0362666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tj_4u2c 2022-08-17T12:52:42.0363616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tj_4u2c/_remote_module_non_scriptable.py 2022-08-17T12:52:42.0364316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi05xazxg 2022-08-17T12:52:42.0367943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi05xazxg/_remote_module_non_scriptable.py 2022-08-17T12:52:42.1952659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:42.2605084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:42.2974127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:42.3221611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:42.5844787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:52:42.5849965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:52:42.5952094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:52:42.5953012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:52:42.5954349Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:42.5955643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:42.5956893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:42.6048815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:43.0291788Z ok (2.578s) 2022-08-17T12:52:43.0292034Z 2022-08-17T12:52:43.0292561Z ---------------------------------------------------------------------- 2022-08-17T12:52:43.0292840Z Ran 1 test in 2.578s 2022-08-17T12:52:43.0292964Z 2022-08-17T12:52:43.0293030Z OK 2022-08-17T12:52:43.0293124Z 2022-08-17T12:52:43.0293219Z Generating XML reports... 2022-08-17T12:52:43.0325606Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125240.xml 2022-08-17T12:52:44.3440795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:44.3441194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:44.3443305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:44.3443733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:44.4963320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1oh4s_6 2022-08-17T12:52:44.4965652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1oh4s_6/_remote_module_non_scriptable.py 2022-08-17T12:52:44.7613682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:44.7624297Z 2022-08-17T12:52:44.7624443Z Running tests... 2022-08-17T12:52:44.7624970Z ---------------------------------------------------------------------- 2022-08-17T12:52:45.0262657Z test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15211 2022-08-17T12:52:45.0293080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15212 2022-08-17T12:52:45.0322149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15213 2022-08-17T12:52:45.0354298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15214 2022-08-17T12:52:46.0620117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:46.0620725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:46.0624032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:46.0624749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:46.1018777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:46.1019490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:46.1022865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:46.1023590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:46.1122537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:46.1123051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:46.1127088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:46.1127762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:46.1248189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:46.1248872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:46.1251754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:46.1252419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:46.2203298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdouo65pd 2022-08-17T12:52:46.2204396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdouo65pd/_remote_module_non_scriptable.py 2022-08-17T12:52:46.2646190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9imge4p 2022-08-17T12:52:46.2648093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9imge4p/_remote_module_non_scriptable.py 2022-08-17T12:52:46.2734819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y09b64a 2022-08-17T12:52:46.2737385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y09b64a/_remote_module_non_scriptable.py 2022-08-17T12:52:46.2829689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqvfnjf0 2022-08-17T12:52:46.2831260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqvfnjf0/_remote_module_non_scriptable.py 2022-08-17T12:52:46.4837711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:46.5393941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:46.5425917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:46.5452093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:46.5701216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:52:46.5802637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:52:46.5822122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:52:46.5822769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:52:46.5823442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:46.5824006Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:46.5904091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:46.5904893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:52:47.4417132Z ok (2.679s) 2022-08-17T12:52:47.4417386Z 2022-08-17T12:52:47.4417859Z ---------------------------------------------------------------------- 2022-08-17T12:52:47.4418188Z Ran 1 test in 2.679s 2022-08-17T12:52:47.4418323Z 2022-08-17T12:52:47.4418389Z OK 2022-08-17T12:52:47.4418488Z 2022-08-17T12:52:47.4418576Z Generating XML reports... 2022-08-17T12:52:47.4452565Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125244.xml 2022-08-17T12:52:48.7435837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:48.7436228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:48.7439077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:48.7439578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:48.8978363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5w6fxkc 2022-08-17T12:52:48.8981304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5w6fxkc/_remote_module_non_scriptable.py 2022-08-17T12:52:49.1652428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:49.1662587Z 2022-08-17T12:52:49.1662745Z Running tests... 2022-08-17T12:52:49.1663353Z ---------------------------------------------------------------------- 2022-08-17T12:52:49.4314469Z test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15610 2022-08-17T12:52:49.4345659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15611 2022-08-17T12:52:49.4378838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15612 2022-08-17T12:52:49.4412631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15613 2022-08-17T12:52:50.4247840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:50.4248536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:50.4251636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:50.4252286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:50.4620926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:50.4621623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:50.4624716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:50.4625367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:50.4751976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:50.4752828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:50.4756107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:50.4756691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:50.4929299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:50.4929992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:50.4933117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:50.4933787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:50.5891874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqivqkvs 2022-08-17T12:52:50.5893196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqivqkvs/_remote_module_non_scriptable.py 2022-08-17T12:52:50.6295495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1oxy5i_x 2022-08-17T12:52:50.6297263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1oxy5i_x/_remote_module_non_scriptable.py 2022-08-17T12:52:50.6381323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhgj4tzv 2022-08-17T12:52:50.6382332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhgj4tzv/_remote_module_non_scriptable.py 2022-08-17T12:52:50.6631210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkidxoc5g 2022-08-17T12:52:50.6632213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkidxoc5g/_remote_module_non_scriptable.py 2022-08-17T12:52:50.8523290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:50.8921461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:50.8978831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:50.9286918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:51.5470782Z ok (2.380s) 2022-08-17T12:52:51.5471007Z 2022-08-17T12:52:51.5471464Z ---------------------------------------------------------------------- 2022-08-17T12:52:51.5471854Z Ran 1 test in 2.381s 2022-08-17T12:52:51.5472032Z 2022-08-17T12:52:51.5472137Z OK 2022-08-17T12:52:51.5472271Z 2022-08-17T12:52:51.5473040Z Generating XML reports... 2022-08-17T12:52:51.5507102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125249.xml 2022-08-17T12:52:52.8541498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:52.8541904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:52.8544920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:52.8545294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:53.0070808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk390vhr8 2022-08-17T12:52:53.0074146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk390vhr8/_remote_module_non_scriptable.py 2022-08-17T12:52:53.2716697Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:53.2728056Z 2022-08-17T12:52:53.2728236Z Running tests... 2022-08-17T12:52:53.2728665Z ---------------------------------------------------------------------- 2022-08-17T12:52:53.5355865Z test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15833 2022-08-17T12:52:53.5386517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15834 2022-08-17T12:52:53.5414891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15835 2022-08-17T12:52:53.5446248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15836 2022-08-17T12:52:54.5735536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:54.5736163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:54.5739281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:54.5739914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:54.6126538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:54.6127243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:54.6132215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:54.6132873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:54.6156928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:54.6157577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:54.6162387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:54.6163037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:54.6319144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:54.6319836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:54.6323067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:54.6323666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:54.7357464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh6keeob 2022-08-17T12:52:54.7358474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh6keeob/_remote_module_non_scriptable.py 2022-08-17T12:52:54.7761395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk68x0fe1 2022-08-17T12:52:54.7764221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk68x0fe1/_remote_module_non_scriptable.py 2022-08-17T12:52:54.7898917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb49tu6b1 2022-08-17T12:52:54.7900030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb49tu6b1/_remote_module_non_scriptable.py 2022-08-17T12:52:54.7944031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk42ez99 2022-08-17T12:52:54.7946071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk42ez99/_remote_module_non_scriptable.py 2022-08-17T12:52:54.9995644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:55.0393837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:55.0541784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:55.0598808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:55.4501504Z ok (2.177s) 2022-08-17T12:52:55.4501740Z 2022-08-17T12:52:55.4502230Z ---------------------------------------------------------------------- 2022-08-17T12:52:55.4502637Z Ran 1 test in 2.177s 2022-08-17T12:52:55.4502829Z 2022-08-17T12:52:55.4502924Z OK 2022-08-17T12:52:55.4503073Z 2022-08-17T12:52:55.4503537Z Generating XML reports... 2022-08-17T12:52:55.4537213Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125253.xml 2022-08-17T12:52:56.7505491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:56.7505879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:56.7508403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:56.7508783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:56.9031613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bcydovd 2022-08-17T12:52:56.9034275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bcydovd/_remote_module_non_scriptable.py 2022-08-17T12:52:57.1692618Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:52:57.1703127Z 2022-08-17T12:52:57.1703257Z Running tests... 2022-08-17T12:52:57.1703704Z ---------------------------------------------------------------------- 2022-08-17T12:52:57.4337258Z test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15892 2022-08-17T12:52:57.4366052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15893 2022-08-17T12:52:57.4394218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15894 2022-08-17T12:52:57.4426209Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15895 2022-08-17T12:52:58.4867860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:58.4868492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:58.4871557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:58.4872228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:58.4999779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:58.5000479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:58.5003793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:58.5004396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:58.5142112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:58.5142767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:58.5146316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:58.5146986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:58.5374333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:52:58.5374896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:52:58.5377902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:52:58.5378432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:52:58.6484980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8s3q9y9 2022-08-17T12:52:58.6485815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8s3q9y9/_remote_module_non_scriptable.py 2022-08-17T12:52:58.6615471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvcgntaj 2022-08-17T12:52:58.6616463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvcgntaj/_remote_module_non_scriptable.py 2022-08-17T12:52:58.6729181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1oy1o3o9 2022-08-17T12:52:58.6729872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1oy1o3o9/_remote_module_non_scriptable.py 2022-08-17T12:52:58.6944526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtrd2u1a 2022-08-17T12:52:58.6945555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtrd2u1a/_remote_module_non_scriptable.py 2022-08-17T12:52:58.9165651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:52:58.9290894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:52:58.9354802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:52:58.9825055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:52:59.6487603Z ok (2.478s) 2022-08-17T12:52:59.6487803Z 2022-08-17T12:52:59.6488149Z ---------------------------------------------------------------------- 2022-08-17T12:52:59.6488416Z Ran 1 test in 2.478s 2022-08-17T12:52:59.6488539Z 2022-08-17T12:52:59.6488603Z OK 2022-08-17T12:52:59.6488961Z 2022-08-17T12:52:59.6489058Z Generating XML reports... 2022-08-17T12:52:59.6521814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125257.xml 2022-08-17T12:53:00.9710686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:00.9711090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:00.9713796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:00.9714178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:01.1250274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzmvpbg0 2022-08-17T12:53:01.1252835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzmvpbg0/_remote_module_non_scriptable.py 2022-08-17T12:53:01.3945309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:01.3956061Z 2022-08-17T12:53:01.3956176Z Running tests... 2022-08-17T12:53:01.3956814Z ---------------------------------------------------------------------- 2022-08-17T12:53:01.6683500Z test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16115 2022-08-17T12:53:01.6714772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16116 2022-08-17T12:53:01.6745053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16117 2022-08-17T12:53:01.6777100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16118 2022-08-17T12:53:02.7010490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:02.7011142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:02.7014465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:02.7015136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:02.7331329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:02.7332003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:02.7334854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:02.7335513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:02.7460783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:02.7461623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:02.7464117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:02.7464921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:02.7685026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:02.7685723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:02.7688543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:02.7689201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:02.8619499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgpp27f6 2022-08-17T12:53:02.8620701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgpp27f6/_remote_module_non_scriptable.py 2022-08-17T12:53:02.8921859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09vli0qj 2022-08-17T12:53:02.8922911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09vli0qj/_remote_module_non_scriptable.py 2022-08-17T12:53:02.9085736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6_2qusv 2022-08-17T12:53:02.9087475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6_2qusv/_remote_module_non_scriptable.py 2022-08-17T12:53:02.9257388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi1hgiw5 2022-08-17T12:53:02.9258182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi1hgiw5/_remote_module_non_scriptable.py 2022-08-17T12:53:03.1483278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:03.1533993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:03.1728327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:03.1888403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:03.4147856Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:03.4149038Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:03.4150640Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:03.4151988Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:03.4152960Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:03.4153824Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:03.7834421Z ok (2.388s) 2022-08-17T12:53:03.7834679Z 2022-08-17T12:53:03.7835236Z ---------------------------------------------------------------------- 2022-08-17T12:53:03.7835513Z Ran 1 test in 2.388s 2022-08-17T12:53:03.7835645Z 2022-08-17T12:53:03.7837341Z OK 2022-08-17T12:53:03.7837910Z 2022-08-17T12:53:03.7838060Z Generating XML reports... 2022-08-17T12:53:03.7869792Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125301.xml 2022-08-17T12:53:05.1009767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:05.1010171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:05.1012610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:05.1013101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:05.2543301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyi7d04_ 2022-08-17T12:53:05.2546441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyi7d04_/_remote_module_non_scriptable.py 2022-08-17T12:53:05.5206386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:05.5217000Z 2022-08-17T12:53:05.5217128Z Running tests... 2022-08-17T12:53:05.5217779Z ---------------------------------------------------------------------- 2022-08-17T12:53:05.7870610Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16338 2022-08-17T12:53:05.7900023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16339 2022-08-17T12:53:05.7929419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16340 2022-08-17T12:53:05.7964559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16341 2022-08-17T12:53:06.8600015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:06.8600686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:06.8603732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:06.8604398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:06.8863883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:06.8864657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:06.8867205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:06.8867796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:06.9142302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:06.9142896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:06.9145388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:06.9145961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:06.9408328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:06.9409025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:06.9411894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:06.9412522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:07.0216043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n0xsnea 2022-08-17T12:53:07.0216836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n0xsnea/_remote_module_non_scriptable.py 2022-08-17T12:53:07.0468970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbxfiz76 2022-08-17T12:53:07.0470066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbxfiz76/_remote_module_non_scriptable.py 2022-08-17T12:53:07.0741170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcry1_uib 2022-08-17T12:53:07.0742254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcry1_uib/_remote_module_non_scriptable.py 2022-08-17T12:53:07.0990430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8hz__6m 2022-08-17T12:53:07.0991266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8hz__6m/_remote_module_non_scriptable.py 2022-08-17T12:53:07.2904025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:07.3122749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:07.3615504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:07.3616085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:07.6190745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:53:07.6350109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:53:07.6350852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:53:07.6354409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:53:07.6356569Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:53:07.6357273Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:53:07.6392863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:53:07.6453254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:53:07.6469500Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:07.6470287Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:07.6471037Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:07.6471762Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:07.6472534Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:07.6473262Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T12:53:07.6484174Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:07.6484897Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T12:53:08.0024780Z ok (2.480s) 2022-08-17T12:53:08.0025014Z 2022-08-17T12:53:08.0025486Z ---------------------------------------------------------------------- 2022-08-17T12:53:08.0026199Z Ran 1 test in 2.481s 2022-08-17T12:53:08.0026321Z 2022-08-17T12:53:08.0026372Z OK 2022-08-17T12:53:08.0026475Z 2022-08-17T12:53:08.0026574Z Generating XML reports... 2022-08-17T12:53:08.0059593Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125305.xml 2022-08-17T12:53:09.3138012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:09.3138609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:09.3139807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:09.3140354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:09.4649739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6dxpwhc 2022-08-17T12:53:09.4653068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6dxpwhc/_remote_module_non_scriptable.py 2022-08-17T12:53:09.7278929Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:09.7289747Z 2022-08-17T12:53:09.7289898Z Running tests... 2022-08-17T12:53:09.7290805Z ---------------------------------------------------------------------- 2022-08-17T12:53:09.9909619Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16573 2022-08-17T12:53:09.9938225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16574 2022-08-17T12:53:09.9966724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16575 2022-08-17T12:53:09.9997490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16576 2022-08-17T12:53:11.0446572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:11.0447247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:11.0450254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:11.0450957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:11.0797196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:11.0797811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:11.0800467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:11.0801061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:11.0882543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:11.0883276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:11.0886914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:11.0887590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:11.1530327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:11.1530921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:11.1533719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:11.1534272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:11.2030935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsl1anao 2022-08-17T12:53:11.2031620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsl1anao/_remote_module_non_scriptable.py 2022-08-17T12:53:11.2378732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcjm7o41 2022-08-17T12:53:11.2379739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcjm7o41/_remote_module_non_scriptable.py 2022-08-17T12:53:11.2487540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3wgppkh 2022-08-17T12:53:11.2488345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3wgppkh/_remote_module_non_scriptable.py 2022-08-17T12:53:11.3131087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwrsfdxw 2022-08-17T12:53:11.3131962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwrsfdxw/_remote_module_non_scriptable.py 2022-08-17T12:53:11.4898588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:11.5009913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:11.5120106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:11.5779233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:12.2058453Z ok (2.477s) 2022-08-17T12:53:12.2058776Z 2022-08-17T12:53:12.2059545Z ---------------------------------------------------------------------- 2022-08-17T12:53:12.2059823Z Ran 1 test in 2.477s 2022-08-17T12:53:12.2059934Z 2022-08-17T12:53:12.2059998Z OK 2022-08-17T12:53:12.2060094Z 2022-08-17T12:53:12.2060197Z Generating XML reports... 2022-08-17T12:53:12.2092748Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125309.xml 2022-08-17T12:53:13.5093448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:13.5093852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:13.5096881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:13.5097345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:13.6598509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdeb9q51v 2022-08-17T12:53:13.6602090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdeb9q51v/_remote_module_non_scriptable.py 2022-08-17T12:53:13.9243041Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:13.9253897Z 2022-08-17T12:53:13.9254014Z Running tests... 2022-08-17T12:53:13.9254463Z ---------------------------------------------------------------------- 2022-08-17T12:53:14.1896814Z test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16796 2022-08-17T12:53:14.1925986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16797 2022-08-17T12:53:14.1955110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16798 2022-08-17T12:53:14.1987179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16799 2022-08-17T12:53:15.3038555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:15.3039190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:15.3041890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:15.3042497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:15.3046668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:15.3047268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:15.3051432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:15.3052094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:15.3080459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:15.3081076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:15.3084798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:15.3085369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:15.3227719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:15.3228394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:15.3232314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:15.3232981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:15.4633312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k_p_r4p 2022-08-17T12:53:15.4634791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7h5nbc7 2022-08-17T12:53:15.4635593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k_p_r4p/_remote_module_non_scriptable.py 2022-08-17T12:53:15.4636342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7h5nbc7/_remote_module_non_scriptable.py 2022-08-17T12:53:15.4659103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk9mj_hy 2022-08-17T12:53:15.4661155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk9mj_hy/_remote_module_non_scriptable.py 2022-08-17T12:53:15.4787734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkojvcft 2022-08-17T12:53:15.4789278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkojvcft/_remote_module_non_scriptable.py 2022-08-17T12:53:15.7223832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:15.7254267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:15.7418370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:15.7484888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:16.4047172Z ok (2.479s) 2022-08-17T12:53:16.4047661Z 2022-08-17T12:53:16.4048278Z ---------------------------------------------------------------------- 2022-08-17T12:53:16.4048776Z Ran 1 test in 2.479s 2022-08-17T12:53:16.4048981Z 2022-08-17T12:53:16.4049050Z OK 2022-08-17T12:53:16.4049134Z 2022-08-17T12:53:16.4049233Z Generating XML reports... 2022-08-17T12:53:16.4084667Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125313.xml 2022-08-17T12:53:17.7021016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:17.7021418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:17.7024205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:17.7024653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:17.8537785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4axio8kh 2022-08-17T12:53:17.8540994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4axio8kh/_remote_module_non_scriptable.py 2022-08-17T12:53:18.1170272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:18.1181157Z 2022-08-17T12:53:18.1181381Z Running tests... 2022-08-17T12:53:18.1181815Z ---------------------------------------------------------------------- 2022-08-17T12:53:18.3794825Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17019 2022-08-17T12:53:18.3822543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17020 2022-08-17T12:53:18.3851924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17021 2022-08-17T12:53:18.3884034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17022 2022-08-17T12:53:19.4619292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:19.4619965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:19.4623114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:19.4623781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:19.4831997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:19.4832671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:19.4836336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:19.4836993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:19.4864707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:19.4865355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:19.4868870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:19.4869559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:19.5006171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:19.5006821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:19.5009384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:19.5010006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:19.6233084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwknwfusz 2022-08-17T12:53:19.6234248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwknwfusz/_remote_module_non_scriptable.py 2022-08-17T12:53:19.6454264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s52wdxs 2022-08-17T12:53:19.6455129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s52wdxs/_remote_module_non_scriptable.py 2022-08-17T12:53:19.6466270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pfc8dce 2022-08-17T12:53:19.6468604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pfc8dce/_remote_module_non_scriptable.py 2022-08-17T12:53:19.6591691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u4tolky 2022-08-17T12:53:19.6592707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u4tolky/_remote_module_non_scriptable.py 2022-08-17T12:53:19.8886921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:19.9110985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:19.9249384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:19.9337747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:20.5944374Z ok (2.476s) 2022-08-17T12:53:20.5944656Z 2022-08-17T12:53:20.5945152Z ---------------------------------------------------------------------- 2022-08-17T12:53:20.5945619Z Ran 1 test in 2.476s 2022-08-17T12:53:20.5945804Z 2022-08-17T12:53:20.5945872Z OK 2022-08-17T12:53:20.5948716Z 2022-08-17T12:53:20.5948975Z Generating XML reports... 2022-08-17T12:53:20.5981000Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125318.xml 2022-08-17T12:53:21.8833649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:21.8834078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:21.8836585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:21.8836960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:22.0359224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvwzk_wt 2022-08-17T12:53:22.0362693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvwzk_wt/_remote_module_non_scriptable.py 2022-08-17T12:53:22.3021731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:22.3031879Z 2022-08-17T12:53:22.3032270Z Running tests... 2022-08-17T12:53:22.3032745Z ---------------------------------------------------------------------- 2022-08-17T12:53:22.5694552Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17242 2022-08-17T12:53:22.5724758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17243 2022-08-17T12:53:22.5753839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17244 2022-08-17T12:53:22.5784335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17245 2022-08-17T12:53:23.5809602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:23.5810226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:23.5811749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:23.5812334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:23.6229950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:23.6230607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:23.6232269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:23.6232903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:23.6287811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:23.6288504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:23.6291723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:23.6292330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:23.6319029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:23.6319567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:23.6332383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:23.6333042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:23.7418508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7niilcn5 2022-08-17T12:53:23.7419487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7niilcn5/_remote_module_non_scriptable.py 2022-08-17T12:53:23.7851764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qsia3f3 2022-08-17T12:53:23.7853032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qsia3f3/_remote_module_non_scriptable.py 2022-08-17T12:53:23.7881845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2un15d15 2022-08-17T12:53:23.7883722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2un15d15/_remote_module_non_scriptable.py 2022-08-17T12:53:23.7926208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdxcy1pz 2022-08-17T12:53:23.7927681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdxcy1pz/_remote_module_non_scriptable.py 2022-08-17T12:53:24.0294281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:24.0486128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:24.0512583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:24.0557169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:24.6843297Z ok (2.381s) 2022-08-17T12:53:24.6843523Z 2022-08-17T12:53:24.6843879Z ---------------------------------------------------------------------- 2022-08-17T12:53:24.6844146Z Ran 1 test in 2.381s 2022-08-17T12:53:24.6844269Z 2022-08-17T12:53:24.6844334Z OK 2022-08-17T12:53:24.6844481Z 2022-08-17T12:53:24.6844574Z Generating XML reports... 2022-08-17T12:53:24.6877503Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125322.xml 2022-08-17T12:53:25.9970525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:25.9970949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:25.9973608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:25.9974061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:26.1518669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyf1xhbh 2022-08-17T12:53:26.1522083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyf1xhbh/_remote_module_non_scriptable.py 2022-08-17T12:53:26.4187338Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:26.4197447Z 2022-08-17T12:53:26.4197547Z Running tests... 2022-08-17T12:53:26.4198373Z ---------------------------------------------------------------------- 2022-08-17T12:53:26.6849353Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17465 2022-08-17T12:53:26.6877191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17466 2022-08-17T12:53:26.6906183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17467 2022-08-17T12:53:26.6937238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17468 2022-08-17T12:53:27.7335729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:27.7336366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:27.7339699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:27.7340365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:27.7732561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:27.7733522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:27.7735300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:27.7735918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:27.7736699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:27.7737379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:27.7741660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:27.7742390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:27.8178587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:27.8179256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:27.8181817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:27.8182543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:27.8914903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mdn9j00 2022-08-17T12:53:27.8915819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mdn9j00/_remote_module_non_scriptable.py 2022-08-17T12:53:27.9327699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5b2udum 2022-08-17T12:53:27.9328491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5b2udum/_remote_module_non_scriptable.py 2022-08-17T12:53:27.9339523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5b46y786 2022-08-17T12:53:27.9341738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5b46y786/_remote_module_non_scriptable.py 2022-08-17T12:53:27.9748098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vkn7gwn 2022-08-17T12:53:27.9748898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vkn7gwn/_remote_module_non_scriptable.py 2022-08-17T12:53:28.1567763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:28.1939877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:28.2215098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:28.2376276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:28.8997370Z ok (2.480s) 2022-08-17T12:53:28.8997660Z 2022-08-17T12:53:28.8998186Z ---------------------------------------------------------------------- 2022-08-17T12:53:28.8998468Z Ran 1 test in 2.480s 2022-08-17T12:53:28.8998595Z 2022-08-17T12:53:28.8998662Z OK 2022-08-17T12:53:28.8998762Z 2022-08-17T12:53:28.8998858Z Generating XML reports... 2022-08-17T12:53:28.9032743Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125326.xml 2022-08-17T12:53:30.2221794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:30.2222200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:30.2224645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:30.2225028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:30.3758087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wzwadt2 2022-08-17T12:53:30.3761560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wzwadt2/_remote_module_non_scriptable.py 2022-08-17T12:53:30.6390046Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:30.6401100Z 2022-08-17T12:53:30.6401541Z Running tests... 2022-08-17T12:53:30.6402032Z ---------------------------------------------------------------------- 2022-08-17T12:53:30.9051418Z test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17688 2022-08-17T12:53:30.9079609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17689 2022-08-17T12:53:30.9109167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17690 2022-08-17T12:53:30.9140077Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17691 2022-08-17T12:53:31.9549424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:31.9550085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:31.9553024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:31.9553688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:31.9746136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:31.9746847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:31.9748367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:31.9748946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:32.0642503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:32.0643128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:32.0646081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:32.0646746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:32.1020507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:32.1021166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:32.1024123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:32.1024838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:32.1135910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn5bw5u8 2022-08-17T12:53:32.1137320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn5bw5u8/_remote_module_non_scriptable.py 2022-08-17T12:53:32.1340271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39_qmfa4 2022-08-17T12:53:32.1341506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39_qmfa4/_remote_module_non_scriptable.py 2022-08-17T12:53:32.2201678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt3c0cnc 2022-08-17T12:53:32.2202481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt3c0cnc/_remote_module_non_scriptable.py 2022-08-17T12:53:32.2596158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nca_4kn 2022-08-17T12:53:32.2597487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nca_4kn/_remote_module_non_scriptable.py 2022-08-17T12:53:32.3781893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:32.4187960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:32.4855005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:32.5212568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:33.1201657Z ok (2.480s) 2022-08-17T12:53:33.1201950Z 2022-08-17T12:53:33.1202484Z ---------------------------------------------------------------------- 2022-08-17T12:53:33.1202967Z Ran 1 test in 2.480s 2022-08-17T12:53:33.1203204Z 2022-08-17T12:53:33.1203283Z OK 2022-08-17T12:53:33.1203423Z 2022-08-17T12:53:33.1203519Z Generating XML reports... 2022-08-17T12:53:33.1237397Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125330.xml 2022-08-17T12:53:34.4219079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:34.4219478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:34.4221917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:34.4222382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:34.5703886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd5806p_ 2022-08-17T12:53:34.5707278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd5806p_/_remote_module_non_scriptable.py 2022-08-17T12:53:34.8307764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:34.8319720Z 2022-08-17T12:53:34.8319840Z Running tests... 2022-08-17T12:53:34.8320489Z ---------------------------------------------------------------------- 2022-08-17T12:53:35.0950498Z test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17911 2022-08-17T12:53:35.0978559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17912 2022-08-17T12:53:35.1007252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17913 2022-08-17T12:53:35.1038515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17914 2022-08-17T12:53:36.1559917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:36.1560619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:36.1563801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:36.1564436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:36.1670291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:36.1670953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:36.1674358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:36.1675062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:36.1773833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:36.1774459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:36.1777920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:36.1778576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:36.2296726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:36.2297274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:36.2300581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:36.2301105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:36.3144346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc30cqr1c 2022-08-17T12:53:36.3145265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc30cqr1c/_remote_module_non_scriptable.py 2022-08-17T12:53:36.3283475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mdzx7f3 2022-08-17T12:53:36.3284600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mdzx7f3/_remote_module_non_scriptable.py 2022-08-17T12:53:36.3360722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yyzvxnh 2022-08-17T12:53:36.3361904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yyzvxnh/_remote_module_non_scriptable.py 2022-08-17T12:53:36.3867900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4myke2s 2022-08-17T12:53:36.3868805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4myke2s/_remote_module_non_scriptable.py 2022-08-17T12:53:36.5767454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:36.5983823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:36.6148673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:36.6471465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:37.4099925Z ok (2.578s) 2022-08-17T12:53:37.4100103Z 2022-08-17T12:53:37.4100611Z ---------------------------------------------------------------------- 2022-08-17T12:53:37.4101080Z Ran 1 test in 2.578s 2022-08-17T12:53:37.4101272Z 2022-08-17T12:53:37.4101337Z OK 2022-08-17T12:53:37.4101433Z 2022-08-17T12:53:37.4101529Z Generating XML reports... 2022-08-17T12:53:37.4133795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125334.xml 2022-08-17T12:53:38.7192393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:38.7192788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:38.7194906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:38.7195303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:38.8686504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbt31hw5 2022-08-17T12:53:38.8689606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbt31hw5/_remote_module_non_scriptable.py 2022-08-17T12:53:39.1322469Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:39.1333377Z 2022-08-17T12:53:39.1333475Z Running tests... 2022-08-17T12:53:39.1334169Z ---------------------------------------------------------------------- 2022-08-17T12:53:39.3956862Z test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18170 2022-08-17T12:53:39.3985191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18171 2022-08-17T12:53:39.4014352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18172 2022-08-17T12:53:39.4047628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18173 2022-08-17T12:53:40.4552580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:40.4553249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:40.4555952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:40.4556652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:40.4940547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:40.4941199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:40.4944482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:40.4945158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:40.5118908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:40.5119472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:40.5121779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:40.5122746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:40.5254360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:40.5255026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:40.5257546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:40.5258475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:40.6168589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphilxm854 2022-08-17T12:53:40.6169402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphilxm854/_remote_module_non_scriptable.py 2022-08-17T12:53:40.6603355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprs2fj9r4 2022-08-17T12:53:40.6604074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprs2fj9r4/_remote_module_non_scriptable.py 2022-08-17T12:53:40.6736837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyzf31ms 2022-08-17T12:53:40.6737779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyzf31ms/_remote_module_non_scriptable.py 2022-08-17T12:53:40.7054034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2a8shlh 2022-08-17T12:53:40.7054817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2a8shlh/_remote_module_non_scriptable.py 2022-08-17T12:53:40.8848277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:40.9250767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:40.9366376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:40.9708398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:41.6106748Z ok (2.477s) 2022-08-17T12:53:41.6106949Z 2022-08-17T12:53:41.6107314Z ---------------------------------------------------------------------- 2022-08-17T12:53:41.6107658Z Ran 1 test in 2.477s 2022-08-17T12:53:41.6107794Z 2022-08-17T12:53:41.6107859Z OK 2022-08-17T12:53:41.6107954Z 2022-08-17T12:53:41.6108050Z Generating XML reports... 2022-08-17T12:53:41.6141331Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125339.xml 2022-08-17T12:53:42.9267279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:42.9267688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:42.9270189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:42.9270558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:43.0764076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdlm23e2 2022-08-17T12:53:43.0767714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdlm23e2/_remote_module_non_scriptable.py 2022-08-17T12:53:43.3369244Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:43.3379876Z 2022-08-17T12:53:43.3379981Z Running tests... 2022-08-17T12:53:43.3380603Z ---------------------------------------------------------------------- 2022-08-17T12:53:43.5952220Z test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18393 2022-08-17T12:53:43.5980975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18394 2022-08-17T12:53:43.6010184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18395 2022-08-17T12:53:43.6041162Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18396 2022-08-17T12:53:44.6402867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:44.6403548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:44.6406521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:44.6407179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:44.6701962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:44.6702667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:44.6705578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:44.6706235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:44.6785427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:44.6786093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:44.6789622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:44.6790262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:44.6791809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:44.6792384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:44.6796884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:44.6797496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:44.8016346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz9x4_58 2022-08-17T12:53:44.8017045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz9x4_58/_remote_module_non_scriptable.py 2022-08-17T12:53:44.8323704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_d_1x2l 2022-08-17T12:53:44.8324959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_d_1x2l/_remote_module_non_scriptable.py 2022-08-17T12:53:44.8401552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoc8k_41 2022-08-17T12:53:44.8402635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoc8k_41/_remote_module_non_scriptable.py 2022-08-17T12:53:44.8574396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposakxa7d 2022-08-17T12:53:44.8575522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposakxa7d/_remote_module_non_scriptable.py 2022-08-17T12:53:45.0623967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:45.0894771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:45.0981653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:45.1158412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:45.8100739Z ok (2.472s) 2022-08-17T12:53:45.8100968Z 2022-08-17T12:53:45.8101394Z ---------------------------------------------------------------------- 2022-08-17T12:53:45.8101751Z Ran 1 test in 2.472s 2022-08-17T12:53:45.8101914Z 2022-08-17T12:53:45.8101997Z OK 2022-08-17T12:53:45.8102124Z 2022-08-17T12:53:45.8102234Z Generating XML reports... 2022-08-17T12:53:45.8145876Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125343.xml 2022-08-17T12:53:47.0946679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:47.0947081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:47.0949517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:47.0949913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:47.2452980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5791tt5w 2022-08-17T12:53:47.2456595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5791tt5w/_remote_module_non_scriptable.py 2022-08-17T12:53:47.5084185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:47.5095392Z 2022-08-17T12:53:47.5095492Z Running tests... 2022-08-17T12:53:47.5095997Z ---------------------------------------------------------------------- 2022-08-17T12:53:47.7717010Z test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18616 2022-08-17T12:53:47.7745677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18617 2022-08-17T12:53:47.7774610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18618 2022-08-17T12:53:47.7805104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18619 2022-08-17T12:53:48.8744151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:48.8744925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:48.8748087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:48.8748718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:48.8922357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:48.8923018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:48.8926260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:48.8926960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:48.9026544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:48.9027249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:48.9031299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:48.9032005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:48.9502725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:48.9503386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:48.9506280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:48.9507211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:49.0405310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dt6_g99 2022-08-17T12:53:49.0406379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dt6_g99/_remote_module_non_scriptable.py 2022-08-17T12:53:49.0547565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgz02ha3p 2022-08-17T12:53:49.0549192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgz02ha3p/_remote_module_non_scriptable.py 2022-08-17T12:53:49.0636439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppg5gp0ol 2022-08-17T12:53:49.0637412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppg5gp0ol/_remote_module_non_scriptable.py 2022-08-17T12:53:49.1064565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr75uzi0r 2022-08-17T12:53:49.1065673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr75uzi0r/_remote_module_non_scriptable.py 2022-08-17T12:53:49.3002726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:49.3139469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:49.3206412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:49.3682220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:50.0867674Z ok (2.577s) 2022-08-17T12:53:50.0867890Z 2022-08-17T12:53:50.0868255Z ---------------------------------------------------------------------- 2022-08-17T12:53:50.0868583Z Ran 1 test in 2.577s 2022-08-17T12:53:50.0868717Z 2022-08-17T12:53:50.0868782Z OK 2022-08-17T12:53:50.0868881Z 2022-08-17T12:53:50.0868977Z Generating XML reports... 2022-08-17T12:53:50.0902822Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125347.xml 2022-08-17T12:53:51.3785268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:51.3785669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:51.3788336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:51.3788938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:51.5298410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0na1uiu6 2022-08-17T12:53:51.5301656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0na1uiu6/_remote_module_non_scriptable.py 2022-08-17T12:53:51.7918472Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:51.7929150Z 2022-08-17T12:53:51.7929256Z Running tests... 2022-08-17T12:53:51.7929826Z ---------------------------------------------------------------------- 2022-08-17T12:53:52.0570659Z test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18839 2022-08-17T12:53:52.0598043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18840 2022-08-17T12:53:52.0628861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18841 2022-08-17T12:53:52.0658281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18842 2022-08-17T12:53:53.1033600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:53.1034232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:53.1036234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:53.1037178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:53.1088818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:53.1089443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:53.1092366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:53.1092950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:53.1364440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:53.1365098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:53.1367935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:53.1368578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:53.1384924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:53.1385544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:53.1389721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:53.1390361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:53.2657792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm58fh6w1 2022-08-17T12:53:53.2658641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm58fh6w1/_remote_module_non_scriptable.py 2022-08-17T12:53:53.2716681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl18i7s98 2022-08-17T12:53:53.2718343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl18i7s98/_remote_module_non_scriptable.py 2022-08-17T12:53:53.2973621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e4556ua 2022-08-17T12:53:53.2974482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e4556ua/_remote_module_non_scriptable.py 2022-08-17T12:53:53.2981603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7k4xztf 2022-08-17T12:53:53.2984240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7k4xztf/_remote_module_non_scriptable.py 2022-08-17T12:53:53.5283023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:53.5364965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:53.5619451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:53.5625093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:54.3719069Z ok (2.579s) 2022-08-17T12:53:54.3719374Z 2022-08-17T12:53:54.3719854Z ---------------------------------------------------------------------- 2022-08-17T12:53:54.3720124Z Ran 1 test in 2.579s 2022-08-17T12:53:54.3720252Z 2022-08-17T12:53:54.3720302Z OK 2022-08-17T12:53:54.3720399Z 2022-08-17T12:53:54.3720495Z Generating XML reports... 2022-08-17T12:53:54.3754590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125351.xml 2022-08-17T12:53:55.6870738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:55.6871138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:55.6873793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:55.6874194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:55.8399973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm6zisvc 2022-08-17T12:53:55.8402742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm6zisvc/_remote_module_non_scriptable.py 2022-08-17T12:53:56.1052855Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:53:56.1063307Z 2022-08-17T12:53:56.1063425Z Running tests... 2022-08-17T12:53:56.1064037Z ---------------------------------------------------------------------- 2022-08-17T12:53:56.3728024Z test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19062 2022-08-17T12:53:56.3755873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19063 2022-08-17T12:53:56.3784943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19064 2022-08-17T12:53:56.3816976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19065 2022-08-17T12:53:57.4204258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:57.4204898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:57.4208483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:57.4209136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:57.4334050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:57.4334720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:57.4338010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:57.4338587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:57.4363414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:57.4364093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:57.4367618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:57.4368298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:57.4472246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:57.4472840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:57.4475777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:57.4476424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:57.5800663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h5y27q8 2022-08-17T12:53:57.5801735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h5y27q8/_remote_module_non_scriptable.py 2022-08-17T12:53:57.5955828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oo1rb1g 2022-08-17T12:53:57.5957090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oo1rb1g/_remote_module_non_scriptable.py 2022-08-17T12:53:57.5963365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3k1uzvs 2022-08-17T12:53:57.5965400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3k1uzvs/_remote_module_non_scriptable.py 2022-08-17T12:53:57.6061137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagzytdt5 2022-08-17T12:53:57.6062949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagzytdt5/_remote_module_non_scriptable.py 2022-08-17T12:53:57.8441369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:53:57.8578246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:53:57.8611340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:53:57.8671151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:53:58.4874320Z ok (2.381s) 2022-08-17T12:53:58.4874552Z 2022-08-17T12:53:58.4874942Z ---------------------------------------------------------------------- 2022-08-17T12:53:58.4875205Z Ran 1 test in 2.381s 2022-08-17T12:53:58.4875331Z 2022-08-17T12:53:58.4875395Z OK 2022-08-17T12:53:58.4875490Z 2022-08-17T12:53:58.4875587Z Generating XML reports... 2022-08-17T12:53:58.4911832Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125356.xml 2022-08-17T12:53:59.7729452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:53:59.7729869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:53:59.7731829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:53:59.7732208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:53:59.9232152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu3qfbdz 2022-08-17T12:53:59.9235280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu3qfbdz/_remote_module_non_scriptable.py 2022-08-17T12:54:00.1863090Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:00.1874939Z 2022-08-17T12:54:00.1875065Z Running tests... 2022-08-17T12:54:00.1875678Z ---------------------------------------------------------------------- 2022-08-17T12:54:00.4535512Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19285 2022-08-17T12:54:00.4563189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19286 2022-08-17T12:54:00.4594628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19287 2022-08-17T12:54:00.4624698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19288 2022-08-17T12:54:01.4782691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:01.4783339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:01.4786297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:01.4786972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:01.5175944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:01.5176580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:01.5179616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:01.5180230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:01.5551079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:01.5551725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:01.5555339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:01.5556047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:01.5882236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:01.5882906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:01.5886565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:01.5887342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:01.6425252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d0t6c5i 2022-08-17T12:54:01.6425923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d0t6c5i/_remote_module_non_scriptable.py 2022-08-17T12:54:01.6760517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n23ozu0 2022-08-17T12:54:01.6761703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n23ozu0/_remote_module_non_scriptable.py 2022-08-17T12:54:01.7157914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xnvl1mh 2022-08-17T12:54:01.7158755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xnvl1mh/_remote_module_non_scriptable.py 2022-08-17T12:54:01.7474243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6zidmbu 2022-08-17T12:54:01.7475029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6zidmbu/_remote_module_non_scriptable.py 2022-08-17T12:54:01.9033283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:01.9382904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:01.9788471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:02.0105213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:02.6684642Z ok (2.481s) 2022-08-17T12:54:02.6684919Z 2022-08-17T12:54:02.6685310Z ---------------------------------------------------------------------- 2022-08-17T12:54:02.6685561Z Ran 1 test in 2.481s 2022-08-17T12:54:02.6687650Z 2022-08-17T12:54:02.6687993Z OK 2022-08-17T12:54:02.6688226Z 2022-08-17T12:54:02.6688338Z Generating XML reports... 2022-08-17T12:54:02.6719675Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125400.xml 2022-08-17T12:54:03.9660564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:03.9661040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:03.9663179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:03.9663554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:04.1192406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44kfh78k 2022-08-17T12:54:04.1195942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44kfh78k/_remote_module_non_scriptable.py 2022-08-17T12:54:04.3882393Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:04.3894005Z 2022-08-17T12:54:04.3894319Z Running tests... 2022-08-17T12:54:04.3894967Z ---------------------------------------------------------------------- 2022-08-17T12:54:04.6587437Z test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19508 2022-08-17T12:54:04.6615269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19509 2022-08-17T12:54:04.6643962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19510 2022-08-17T12:54:04.6683404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19511 2022-08-17T12:54:05.7020812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:05.7021444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:05.7024591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:05.7025276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:05.7081638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:05.7082251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:05.7085120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:05.7085745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:05.7107383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:05.7107967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:05.7111732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:05.7112361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:05.7224774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:05.7225714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:05.7228720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:05.7229384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:05.8636125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5a8c0d4 2022-08-17T12:54:05.8636862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5a8c0d4/_remote_module_non_scriptable.py 2022-08-17T12:54:05.8694405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeop_aeh4 2022-08-17T12:54:05.8695705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeop_aeh4/_remote_module_non_scriptable.py 2022-08-17T12:54:05.8697616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_b4fk_d0 2022-08-17T12:54:05.8700595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_b4fk_d0/_remote_module_non_scriptable.py 2022-08-17T12:54:05.8815883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq7wvjuw 2022-08-17T12:54:05.8817121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq7wvjuw/_remote_module_non_scriptable.py 2022-08-17T12:54:06.1248665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:06.1275068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:06.1304006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:06.1407695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:06.7764752Z ok (2.384s) 2022-08-17T12:54:06.7765022Z 2022-08-17T12:54:06.7765557Z ---------------------------------------------------------------------- 2022-08-17T12:54:06.7766167Z Ran 1 test in 2.385s 2022-08-17T12:54:06.7766388Z 2022-08-17T12:54:06.7766530Z OK 2022-08-17T12:54:06.7766681Z 2022-08-17T12:54:06.7766792Z Generating XML reports... 2022-08-17T12:54:06.7785168Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125404.xml 2022-08-17T12:54:08.0613982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:08.0614384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:08.0616637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:08.0617245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:08.2141789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6ya98hv 2022-08-17T12:54:08.2145706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6ya98hv/_remote_module_non_scriptable.py 2022-08-17T12:54:08.4783338Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:08.4794409Z 2022-08-17T12:54:08.4794872Z Running tests... 2022-08-17T12:54:08.4795284Z ---------------------------------------------------------------------- 2022-08-17T12:54:08.7422546Z test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19731 2022-08-17T12:54:08.7450997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19732 2022-08-17T12:54:08.7481668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19733 2022-08-17T12:54:08.7522015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19734 2022-08-17T12:54:09.8124448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:09.8125040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:09.8128008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:09.8128622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:09.8165560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:09.8166350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:09.8171248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:09.8171946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:09.8725179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:09.8726579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:09.8727773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:09.8728333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:09.8729155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:09.8729833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:09.8733144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:09.8733716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:09.9724838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp7_iuc5 2022-08-17T12:54:09.9725751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp7_iuc5/_remote_module_non_scriptable.py 2022-08-17T12:54:09.9864564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshj9ir15 2022-08-17T12:54:09.9867169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshj9ir15/_remote_module_non_scriptable.py 2022-08-17T12:54:10.0305289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqc9_l4_ 2022-08-17T12:54:10.0306361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqc9_l4_/_remote_module_non_scriptable.py 2022-08-17T12:54:10.0316393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfx_yvv_ 2022-08-17T12:54:10.0318122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfx_yvv_/_remote_module_non_scriptable.py 2022-08-17T12:54:10.2304889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:10.2498119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:10.2903671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:10.2927520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:10.9581509Z ok (2.478s) 2022-08-17T12:54:10.9581797Z 2022-08-17T12:54:10.9582298Z ---------------------------------------------------------------------- 2022-08-17T12:54:10.9582551Z Ran 1 test in 2.479s 2022-08-17T12:54:10.9582675Z 2022-08-17T12:54:10.9582740Z OK 2022-08-17T12:54:10.9582838Z 2022-08-17T12:54:10.9582936Z Generating XML reports... 2022-08-17T12:54:10.9616411Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125408.xml 2022-08-17T12:54:12.2606118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:12.2606558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:12.2610028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:12.2610417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:12.4147612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6teusu2n 2022-08-17T12:54:12.4150447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6teusu2n/_remote_module_non_scriptable.py 2022-08-17T12:54:12.6810726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:12.6821332Z 2022-08-17T12:54:12.6821647Z Running tests... 2022-08-17T12:54:12.6822025Z ---------------------------------------------------------------------- 2022-08-17T12:54:12.9456681Z test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19954 2022-08-17T12:54:12.9484573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19955 2022-08-17T12:54:12.9515150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19956 2022-08-17T12:54:12.9545023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19957 2022-08-17T12:54:14.0280359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:14.0280965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:14.0284038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:14.0434454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:14.0435400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:14.0436024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:14.0438644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:14.0439293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:14.0588130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:14.0588740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:14.0592165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:14.0592873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:14.0869643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:14.0870606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:14.0873318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:14.0873910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:14.1873776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0w1x_h6 2022-08-17T12:54:14.1874788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0w1x_h6/_remote_module_non_scriptable.py 2022-08-17T12:54:14.2011257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp912ecw6j 2022-08-17T12:54:14.2012460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp912ecw6j/_remote_module_non_scriptable.py 2022-08-17T12:54:14.2178835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp811yngdj 2022-08-17T12:54:14.2179673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp811yngdj/_remote_module_non_scriptable.py 2022-08-17T12:54:14.2434369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7g94_8p 2022-08-17T12:54:14.2435747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7g94_8p/_remote_module_non_scriptable.py 2022-08-17T12:54:14.4467803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:14.4598975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:14.4774514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:14.5034176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:15.1604271Z ok (2.478s) 2022-08-17T12:54:15.1604488Z 2022-08-17T12:54:15.1604915Z ---------------------------------------------------------------------- 2022-08-17T12:54:15.1605194Z Ran 1 test in 2.478s 2022-08-17T12:54:15.1605317Z 2022-08-17T12:54:15.1605380Z OK 2022-08-17T12:54:15.1605543Z 2022-08-17T12:54:15.1605642Z Generating XML reports... 2022-08-17T12:54:15.1641455Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125412.xml 2022-08-17T12:54:16.4674044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:16.4674436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:16.4676697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:16.4677057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:16.6200534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe80llila 2022-08-17T12:54:16.6204364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe80llila/_remote_module_non_scriptable.py 2022-08-17T12:54:16.8847398Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:16.8857873Z 2022-08-17T12:54:16.8858100Z Running tests... 2022-08-17T12:54:16.8858739Z ---------------------------------------------------------------------- 2022-08-17T12:54:17.1478329Z test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20197 2022-08-17T12:54:17.1505800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20198 2022-08-17T12:54:17.1536850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20199 2022-08-17T12:54:17.1566064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20200 2022-08-17T12:54:18.2378774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:18.2379753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:18.2382647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:18.2383287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:18.2395452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:18.2396039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:18.2399702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:18.2400375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:18.2426609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:18.2427244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:18.2431618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:18.2432194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:18.2514395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:18.2515076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:18.2518106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:18.2518756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:18.3987619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsgxvn_j 2022-08-17T12:54:18.3988462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsgxvn_j/_remote_module_non_scriptable.py 2022-08-17T12:54:18.3994819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38b6v_ng 2022-08-17T12:54:18.3996799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38b6v_ng/_remote_module_non_scriptable.py 2022-08-17T12:54:18.4036988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d_yj33c 2022-08-17T12:54:18.4039338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d_yj33c/_remote_module_non_scriptable.py 2022-08-17T12:54:18.4097022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n2z_f46 2022-08-17T12:54:18.4098919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n2z_f46/_remote_module_non_scriptable.py 2022-08-17T12:54:18.6597405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:18.6600880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:18.6667078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:18.6707209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:19.3626767Z ok (2.477s) 2022-08-17T12:54:19.3627069Z 2022-08-17T12:54:19.3627613Z ---------------------------------------------------------------------- 2022-08-17T12:54:19.3627869Z Ran 1 test in 2.477s 2022-08-17T12:54:19.3627993Z 2022-08-17T12:54:19.3628059Z OK 2022-08-17T12:54:19.3628155Z 2022-08-17T12:54:19.3628251Z Generating XML reports... 2022-08-17T12:54:19.3661230Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125416.xml 2022-08-17T12:54:20.6790100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:20.6790498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:20.6792094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:20.6792703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:20.8310228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8dlz8qs 2022-08-17T12:54:20.8313560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8dlz8qs/_remote_module_non_scriptable.py 2022-08-17T12:54:21.0943237Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:21.0953847Z 2022-08-17T12:54:21.0953940Z Running tests... 2022-08-17T12:54:21.0954904Z ---------------------------------------------------------------------- 2022-08-17T12:54:21.3621821Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20420 2022-08-17T12:54:21.3650831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20421 2022-08-17T12:54:21.3679511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20422 2022-08-17T12:54:21.3711721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20423 2022-08-17T12:54:22.4000362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:22.4003398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:22.4004348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:22.4004979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:22.4193885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:22.4194619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:22.4198313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:22.4198985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:22.4389445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:22.4390142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:22.4393238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:22.4393892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:22.4494200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:22.4494888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:22.4498223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:22.4498909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:22.5612786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc86279tn 2022-08-17T12:54:22.5614088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc86279tn/_remote_module_non_scriptable.py 2022-08-17T12:54:22.5796390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tsw2nf7 2022-08-17T12:54:22.5797137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tsw2nf7/_remote_module_non_scriptable.py 2022-08-17T12:54:22.6000497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiar8mnxq 2022-08-17T12:54:22.6001488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiar8mnxq/_remote_module_non_scriptable.py 2022-08-17T12:54:22.6105316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkve0yvxs 2022-08-17T12:54:22.6106802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkve0yvxs/_remote_module_non_scriptable.py 2022-08-17T12:54:22.8264161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:22.8419792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:22.8625442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:22.8747780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:22.9012745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:54:22.9112761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:54:22.9214894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:54:22.9215630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:54:22.9216492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:54:22.9217334Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:54:22.9217898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:54:22.9218448Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:54:23.2764870Z ok (2.181s) 2022-08-17T12:54:23.2765046Z 2022-08-17T12:54:23.2765487Z ---------------------------------------------------------------------- 2022-08-17T12:54:23.2766068Z Ran 1 test in 2.181s 2022-08-17T12:54:23.2766287Z 2022-08-17T12:54:23.2766402Z OK 2022-08-17T12:54:23.2766573Z 2022-08-17T12:54:23.2766732Z Generating XML reports... 2022-08-17T12:54:23.2798991Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125421.xml 2022-08-17T12:54:24.6113443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:24.6113841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:24.6116426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:24.6116942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:24.7633973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc85gpxdc 2022-08-17T12:54:24.7637237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc85gpxdc/_remote_module_non_scriptable.py 2022-08-17T12:54:25.0278895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:25.0289687Z 2022-08-17T12:54:25.0290109Z Running tests... 2022-08-17T12:54:25.0290511Z ---------------------------------------------------------------------- 2022-08-17T12:54:25.2918714Z test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20491 2022-08-17T12:54:25.2947580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20492 2022-08-17T12:54:25.2976882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20493 2022-08-17T12:54:25.3009952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20494 2022-08-17T12:54:26.3756100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:26.3756717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:26.3758989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:26.3760374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:26.3777777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:26.3778452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:26.3782341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:26.3782951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:26.3992353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:26.3992990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:26.3996414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:26.3997110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:26.4060610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:26.4061537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:26.4065056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:26.4065702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:26.5380632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty6du40f 2022-08-17T12:54:26.5382080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty6du40f/_remote_module_non_scriptable.py 2022-08-17T12:54:26.5501603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphns2oswj 2022-08-17T12:54:26.5502834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphns2oswj/_remote_module_non_scriptable.py 2022-08-17T12:54:26.5642858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7achjodg 2022-08-17T12:54:26.5644757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7achjodg/_remote_module_non_scriptable.py 2022-08-17T12:54:26.5665059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl99om56h 2022-08-17T12:54:26.5667249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl99om56h/_remote_module_non_scriptable.py 2022-08-17T12:54:26.8156743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:26.8165394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:26.8309062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:26.8385442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:27.5070889Z ok (2.478s) 2022-08-17T12:54:27.5071114Z 2022-08-17T12:54:27.5071571Z ---------------------------------------------------------------------- 2022-08-17T12:54:27.5071837Z Ran 1 test in 2.478s 2022-08-17T12:54:27.5071959Z 2022-08-17T12:54:27.5072022Z OK 2022-08-17T12:54:27.5072120Z 2022-08-17T12:54:27.5072215Z Generating XML reports... 2022-08-17T12:54:27.5106054Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125425.xml 2022-08-17T12:54:28.8245651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:28.8246237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:28.8248450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:28.8249015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:28.9774872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bpnpoae 2022-08-17T12:54:28.9778724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bpnpoae/_remote_module_non_scriptable.py 2022-08-17T12:54:29.2443214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:29.2456969Z 2022-08-17T12:54:29.2457455Z Running tests... 2022-08-17T12:54:29.2457878Z ---------------------------------------------------------------------- 2022-08-17T12:54:29.5141088Z test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20714 2022-08-17T12:54:29.5171588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20715 2022-08-17T12:54:29.5201404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20716 2022-08-17T12:54:29.5234995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20717 2022-08-17T12:54:30.6178890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:30.6179518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:30.6182911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:30.6183564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:30.6366491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:30.6367297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:30.6368192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:30.6368846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:30.6743025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:30.6743620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:30.6746720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:30.6747309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:30.6791381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:30.6792065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:30.6796438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:30.6797152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:30.7969073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeepf_zzj 2022-08-17T12:54:30.7969878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzwzssqv 2022-08-17T12:54:30.7971046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeepf_zzj/_remote_module_non_scriptable.py 2022-08-17T12:54:30.7971837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzwzssqv/_remote_module_non_scriptable.py 2022-08-17T12:54:30.8412250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxlloc5t 2022-08-17T12:54:30.8413067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxlloc5t/_remote_module_non_scriptable.py 2022-08-17T12:54:30.8423548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcr04uw3s 2022-08-17T12:54:30.8425169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcr04uw3s/_remote_module_non_scriptable.py 2022-08-17T12:54:31.0598353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:31.0608023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:31.1037873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:31.1060955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:31.7295644Z ok (2.484s) 2022-08-17T12:54:31.7295916Z 2022-08-17T12:54:31.7296423Z ---------------------------------------------------------------------- 2022-08-17T12:54:31.7296910Z Ran 1 test in 2.484s 2022-08-17T12:54:31.7297081Z 2022-08-17T12:54:31.7297146Z OK 2022-08-17T12:54:31.7297242Z 2022-08-17T12:54:31.7297341Z Generating XML reports... 2022-08-17T12:54:31.7364343Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125429.xml 2022-08-17T12:54:33.0277444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:33.0277894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:33.0280511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:33.0281127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:33.1797330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltdrgg7c 2022-08-17T12:54:33.1800605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltdrgg7c/_remote_module_non_scriptable.py 2022-08-17T12:54:33.4451750Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:33.4462469Z 2022-08-17T12:54:33.4462697Z Running tests... 2022-08-17T12:54:33.4463104Z ---------------------------------------------------------------------- 2022-08-17T12:54:33.7099475Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20937 2022-08-17T12:54:33.7128470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20938 2022-08-17T12:54:33.7157167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20939 2022-08-17T12:54:33.7188854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20940 2022-08-17T12:54:34.7558516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:34.7559187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:34.7562101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:34.7562777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:34.7570502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:34.7571143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:34.7576136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:34.7576823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:34.7678770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:34.7679429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:34.7682548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:34.7683175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:34.7759278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:34.7760197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:34.7763549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:34.7764235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:34.9193539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77vgm2su 2022-08-17T12:54:34.9194649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77vgm2su/_remote_module_non_scriptable.py 2022-08-17T12:54:34.9280897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkm4ajyk 2022-08-17T12:54:34.9282199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkm4ajyk/_remote_module_non_scriptable.py 2022-08-17T12:54:34.9355435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujmyx3un 2022-08-17T12:54:34.9356349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy6txjal 2022-08-17T12:54:34.9357124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujmyx3un/_remote_module_non_scriptable.py 2022-08-17T12:54:34.9359590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy6txjal/_remote_module_non_scriptable.py 2022-08-17T12:54:35.1787283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:35.1849293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:35.1969737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:35.1970368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:35.8245732Z ok (2.378s) 2022-08-17T12:54:35.8246109Z 2022-08-17T12:54:35.8246653Z ---------------------------------------------------------------------- 2022-08-17T12:54:35.8247059Z Ran 1 test in 2.378s 2022-08-17T12:54:35.8247183Z 2022-08-17T12:54:35.8247247Z OK 2022-08-17T12:54:35.8247330Z 2022-08-17T12:54:35.8247427Z Generating XML reports... 2022-08-17T12:54:35.8282339Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125433.xml 2022-08-17T12:54:37.1243367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:37.1243763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:37.1246638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:37.1247134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:37.2744430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe09rozvm 2022-08-17T12:54:37.2747284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe09rozvm/_remote_module_non_scriptable.py 2022-08-17T12:54:37.5376906Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:37.5387921Z 2022-08-17T12:54:37.5388067Z Running tests... 2022-08-17T12:54:37.5388706Z ---------------------------------------------------------------------- 2022-08-17T12:54:37.8062371Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21160 2022-08-17T12:54:37.8091318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21161 2022-08-17T12:54:37.8120710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21162 2022-08-17T12:54:37.8152032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21163 2022-08-17T12:54:38.8512553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:38.8513484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:38.8515955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:38.8516560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:38.8818028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:38.8818665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:38.8821069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:38.8821687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:38.8985393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:38.8986081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:38.8989763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:38.8990347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:38.9649140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:38.9649851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:38.9652670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:38.9653377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:39.0116688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ki51p8o 2022-08-17T12:54:39.0117571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ki51p8o/_remote_module_non_scriptable.py 2022-08-17T12:54:39.0581110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhrsezf1 2022-08-17T12:54:39.0582732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhrsezf1/_remote_module_non_scriptable.py 2022-08-17T12:54:39.0734955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi39d6rk1 2022-08-17T12:54:39.0735573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi39d6rk1/_remote_module_non_scriptable.py 2022-08-17T12:54:39.1225453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax0yu9os 2022-08-17T12:54:39.1226727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax0yu9os/_remote_module_non_scriptable.py 2022-08-17T12:54:39.3020336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:39.3247357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:39.3334971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:39.3826273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:40.0213777Z ok (2.482s) 2022-08-17T12:54:40.0214011Z 2022-08-17T12:54:40.0214394Z ---------------------------------------------------------------------- 2022-08-17T12:54:40.0214688Z Ran 1 test in 2.482s 2022-08-17T12:54:40.0214855Z 2022-08-17T12:54:40.0214920Z OK 2022-08-17T12:54:40.0215015Z 2022-08-17T12:54:40.0215113Z Generating XML reports... 2022-08-17T12:54:40.0247985Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125437.xml 2022-08-17T12:54:41.3184289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:41.3184760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:41.3187162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:41.3187814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:41.4731072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr0nsi5_ 2022-08-17T12:54:41.4733979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr0nsi5_/_remote_module_non_scriptable.py 2022-08-17T12:54:41.7380611Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:41.7391526Z 2022-08-17T12:54:41.7391626Z Running tests... 2022-08-17T12:54:41.7392537Z ---------------------------------------------------------------------- 2022-08-17T12:54:42.0046771Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21383 2022-08-17T12:54:42.0076347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21384 2022-08-17T12:54:42.0105041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21385 2022-08-17T12:54:42.0137222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21386 2022-08-17T12:54:43.0822078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:43.0822791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:43.0825208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:43.0825886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:43.0911447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:43.0912134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:43.0916227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:43.0916899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:43.1060762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:43.1061452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:43.1065355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:43.1066021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:43.1151373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:43.1152040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:43.1155677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:43.1156368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:43.2461978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ago8ao6 2022-08-17T12:54:43.2462894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ago8ao6/_remote_module_non_scriptable.py 2022-08-17T12:54:43.2509631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1whbzei 2022-08-17T12:54:43.2511301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1whbzei/_remote_module_non_scriptable.py 2022-08-17T12:54:43.2647197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasmyypf0 2022-08-17T12:54:43.2648393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasmyypf0/_remote_module_non_scriptable.py 2022-08-17T12:54:43.2921326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jk5di3h 2022-08-17T12:54:43.2922354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jk5di3h/_remote_module_non_scriptable.py 2022-08-17T12:54:43.5082059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:43.5085770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:43.5253227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:43.5537832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:44.2199586Z ok (2.480s) 2022-08-17T12:54:44.2199856Z 2022-08-17T12:54:44.2200387Z ---------------------------------------------------------------------- 2022-08-17T12:54:44.2200790Z Ran 1 test in 2.481s 2022-08-17T12:54:44.2200911Z 2022-08-17T12:54:44.2200974Z OK 2022-08-17T12:54:44.2201071Z 2022-08-17T12:54:44.2201166Z Generating XML reports... 2022-08-17T12:54:44.2286708Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125441.xml 2022-08-17T12:54:45.5269415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:45.5269805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:45.5272445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:45.5272814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:45.6795214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvpse5u3 2022-08-17T12:54:45.6798582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvpse5u3/_remote_module_non_scriptable.py 2022-08-17T12:54:45.9418167Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:45.9428654Z 2022-08-17T12:54:45.9428822Z Running tests... 2022-08-17T12:54:45.9429275Z ---------------------------------------------------------------------- 2022-08-17T12:54:46.2038176Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21612 2022-08-17T12:54:46.2066897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21613 2022-08-17T12:54:46.2095052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21614 2022-08-17T12:54:46.2125451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21615 2022-08-17T12:54:47.2801088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:47.2801722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:47.2804864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:47.2805562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:47.2812044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:47.2812721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:47.2816433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:47.2817035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:47.3338691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:47.3339351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:47.3342419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:47.3343287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:47.3381438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:47.3382063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:47.3387000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:47.3387702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:47.4441044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngrrq23x 2022-08-17T12:54:47.4441844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngrrq23x/_remote_module_non_scriptable.py 2022-08-17T12:54:47.4612188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_z3vfn9 2022-08-17T12:54:47.4613297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_z3vfn9/_remote_module_non_scriptable.py 2022-08-17T12:54:47.4900919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw7j73rm 2022-08-17T12:54:47.4902347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw7j73rm/_remote_module_non_scriptable.py 2022-08-17T12:54:47.4960077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalae49by 2022-08-17T12:54:47.4961003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalae49by/_remote_module_non_scriptable.py 2022-08-17T12:54:47.7049693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:47.7236679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:47.7479606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:47.7557372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:48.4185602Z ok (2.475s) 2022-08-17T12:54:48.4185869Z 2022-08-17T12:54:48.4186428Z ---------------------------------------------------------------------- 2022-08-17T12:54:48.4186820Z Ran 1 test in 2.476s 2022-08-17T12:54:48.4186945Z 2022-08-17T12:54:48.4186999Z OK 2022-08-17T12:54:48.4187097Z 2022-08-17T12:54:48.4187195Z Generating XML reports... 2022-08-17T12:54:48.4220683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125445.xml 2022-08-17T12:54:49.7111315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:49.7111712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:49.7114171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:49.7114555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:49.8614680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qp79d6n 2022-08-17T12:54:49.8618466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qp79d6n/_remote_module_non_scriptable.py 2022-08-17T12:54:50.1220467Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:50.1231647Z 2022-08-17T12:54:50.1231993Z Running tests... 2022-08-17T12:54:50.1232432Z ---------------------------------------------------------------------- 2022-08-17T12:54:50.3808627Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21835 2022-08-17T12:54:50.3836689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21836 2022-08-17T12:54:50.3864903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21837 2022-08-17T12:54:50.3896100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21838 2022-08-17T12:54:51.4109654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:51.4110350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:51.4113053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:51.4113764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:51.4280672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:51.4281307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:51.4284719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:51.4285371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:51.4506868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:51.4507581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:51.4511799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:51.4512823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:51.4588461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:51.4588959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:51.4593242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:51.4593898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:51.5709063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uxbg87l 2022-08-17T12:54:51.5709915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uxbg87l/_remote_module_non_scriptable.py 2022-08-17T12:54:51.5876734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c1gjshz 2022-08-17T12:54:51.5877505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c1gjshz/_remote_module_non_scriptable.py 2022-08-17T12:54:51.6174474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_jkelrw 2022-08-17T12:54:51.6176188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_jkelrw/_remote_module_non_scriptable.py 2022-08-17T12:54:51.6305409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjej8wam0 2022-08-17T12:54:51.6306602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjej8wam0/_remote_module_non_scriptable.py 2022-08-17T12:54:51.8328167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:51.8479159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:51.8807507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:51.8882008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:52.4954179Z ok (2.372s) 2022-08-17T12:54:52.4954395Z 2022-08-17T12:54:52.4954786Z ---------------------------------------------------------------------- 2022-08-17T12:54:52.4955039Z Ran 1 test in 2.372s 2022-08-17T12:54:52.4955162Z 2022-08-17T12:54:52.4955227Z OK 2022-08-17T12:54:52.4955326Z 2022-08-17T12:54:52.4955483Z Generating XML reports... 2022-08-17T12:54:52.4988793Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125450.xml 2022-08-17T12:54:53.7993450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:53.7994113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:53.7996656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:53.7997081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:53.9521434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfr6kxwd 2022-08-17T12:54:53.9524442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfr6kxwd/_remote_module_non_scriptable.py 2022-08-17T12:54:54.2174128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:54.2186123Z 2022-08-17T12:54:54.2186264Z Running tests... 2022-08-17T12:54:54.2186745Z ---------------------------------------------------------------------- 2022-08-17T12:54:54.4828795Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22058 2022-08-17T12:54:54.4860059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22059 2022-08-17T12:54:54.4888880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22060 2022-08-17T12:54:54.4922917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22061 2022-08-17T12:54:55.5114152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:55.5114829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:55.5116600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:55.5117247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:55.5349205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:55.5349855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:55.5352948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:55.5353614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:55.5722479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:55.5723129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:55.5726814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:55.5727463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:55.5785964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:55.5786596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:55.5790117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:55.6697299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:55.6698071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmklo8kgx 2022-08-17T12:54:55.6698865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmklo8kgx/_remote_module_non_scriptable.py 2022-08-17T12:54:55.7004616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaymqpev6 2022-08-17T12:54:55.7005986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaymqpev6/_remote_module_non_scriptable.py 2022-08-17T12:54:55.7373063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj6bgwkq 2022-08-17T12:54:55.7374150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj6bgwkq/_remote_module_non_scriptable.py 2022-08-17T12:54:55.7508109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuesbs58 2022-08-17T12:54:55.7509199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuesbs58/_remote_module_non_scriptable.py 2022-08-17T12:54:55.9318267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:54:55.9634823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:54:56.0031389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:54:56.0141700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:54:56.6982423Z ok (2.479s) 2022-08-17T12:54:56.6982586Z 2022-08-17T12:54:56.6982945Z ---------------------------------------------------------------------- 2022-08-17T12:54:56.6983220Z Ran 1 test in 2.480s 2022-08-17T12:54:56.6983353Z 2022-08-17T12:54:56.6983435Z OK 2022-08-17T12:54:56.6983599Z 2022-08-17T12:54:56.6983694Z Generating XML reports... 2022-08-17T12:54:56.7016911Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125454.xml 2022-08-17T12:54:57.9881849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:57.9882277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:57.9884786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:57.9885148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:58.1385394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wz0lmnd 2022-08-17T12:54:58.1388644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wz0lmnd/_remote_module_non_scriptable.py 2022-08-17T12:54:58.4002655Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:54:58.4014540Z 2022-08-17T12:54:58.4014679Z Running tests... 2022-08-17T12:54:58.4015283Z ---------------------------------------------------------------------- 2022-08-17T12:54:58.6594332Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22281 2022-08-17T12:54:58.6621436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22282 2022-08-17T12:54:58.6648874Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22283 2022-08-17T12:54:58.6680699Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22284 2022-08-17T12:54:59.7230417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:59.7231087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:59.7233971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:59.7234642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:59.7284528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:59.7285140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:59.7288844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:59.7289468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:59.7320593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:59.7321183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:59.7326246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:59.7327229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:59.7393816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:54:59.7394442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:54:59.7398222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:54:59.7398898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:54:59.8836545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc7_g35i 2022-08-17T12:54:59.8837304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc7_g35i/_remote_module_non_scriptable.py 2022-08-17T12:54:59.8876719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhzgmsc0 2022-08-17T12:54:59.8878197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhzgmsc0/_remote_module_non_scriptable.py 2022-08-17T12:54:59.8927360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8htqhrru 2022-08-17T12:54:59.8929893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8htqhrru/_remote_module_non_scriptable.py 2022-08-17T12:54:59.9186675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv1e92sk 2022-08-17T12:54:59.9187692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv1e92sk/_remote_module_non_scriptable.py 2022-08-17T12:55:00.1450492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:00.1470918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:00.1553893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:00.1799761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:00.8740080Z ok (2.472s) 2022-08-17T12:55:00.8740318Z 2022-08-17T12:55:00.8740792Z ---------------------------------------------------------------------- 2022-08-17T12:55:00.8741219Z Ran 1 test in 2.472s 2022-08-17T12:55:00.8741419Z 2022-08-17T12:55:00.8741514Z OK 2022-08-17T12:55:00.8741812Z 2022-08-17T12:55:00.8741959Z Generating XML reports... 2022-08-17T12:55:00.8774725Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125458.xml 2022-08-17T12:55:02.1813124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:02.1813526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:02.1816049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:02.1816609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:02.3357264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8n0z6j5 2022-08-17T12:55:02.3360405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8n0z6j5/_remote_module_non_scriptable.py 2022-08-17T12:55:02.6026438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:02.6036935Z 2022-08-17T12:55:02.6037038Z Running tests... 2022-08-17T12:55:02.6037648Z ---------------------------------------------------------------------- 2022-08-17T12:55:02.8696844Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22504 2022-08-17T12:55:02.8726673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22505 2022-08-17T12:55:02.8757888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22506 2022-08-17T12:55:02.8797790Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22507 2022-08-17T12:55:03.9406622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:03.9407441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:03.9410487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:03.9411144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:03.9789328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:03.9789972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:03.9791713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:03.9792335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:04.0098664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:04.0099326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:04.0102719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:04.0103430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:04.0259567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:04.0260228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:04.0263025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:04.0263727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:04.1020712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hofiv5e 2022-08-17T12:55:04.1022029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hofiv5e/_remote_module_non_scriptable.py 2022-08-17T12:55:04.1374229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmb2jaxcg 2022-08-17T12:55:04.1375092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmb2jaxcg/_remote_module_non_scriptable.py 2022-08-17T12:55:04.1668372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00oles96 2022-08-17T12:55:04.1669208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00oles96/_remote_module_non_scriptable.py 2022-08-17T12:55:04.1827885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz87f649 2022-08-17T12:55:04.1828880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz87f649/_remote_module_non_scriptable.py 2022-08-17T12:55:04.3660730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:04.3967184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:04.4237288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:04.4428006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:06.6954216Z STAGE:2022-08-17 12:55:06 22505:22505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T12:55:06.6955575Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-08-17T12:55:07.6987199Z STAGE:2022-08-17 12:55:07 22505:22505 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T12:55:08.0905292Z ok (5.486s) 2022-08-17T12:55:08.0905873Z 2022-08-17T12:55:08.0906383Z ---------------------------------------------------------------------- 2022-08-17T12:55:08.0906634Z Ran 1 test in 5.487s 2022-08-17T12:55:08.0906756Z 2022-08-17T12:55:08.0906822Z OK 2022-08-17T12:55:08.0906923Z 2022-08-17T12:55:08.0907020Z Generating XML reports... 2022-08-17T12:55:08.0939795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125502.xml 2022-08-17T12:55:09.3718037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:09.3718416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:09.3721188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:09.3721569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:09.5205415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk31aemm2 2022-08-17T12:55:09.5208717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk31aemm2/_remote_module_non_scriptable.py 2022-08-17T12:55:09.7837767Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:09.7848381Z 2022-08-17T12:55:09.7848736Z Running tests... 2022-08-17T12:55:09.7849095Z ---------------------------------------------------------------------- 2022-08-17T12:55:10.0481223Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22728 2022-08-17T12:55:10.0511716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22729 2022-08-17T12:55:10.0540436Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22730 2022-08-17T12:55:10.0570903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22731 2022-08-17T12:55:11.1304197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:11.1304912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:11.1308010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:11.1308659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:11.1380246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:11.1380918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:11.1384665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:11.1385450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:11.1386242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:11.1386807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:11.1390056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:11.1390729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:11.1649720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:11.1650359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:11.1652629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:11.1653276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:11.2912115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3si_4dv 2022-08-17T12:55:11.2913583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3si_4dv/_remote_module_non_scriptable.py 2022-08-17T12:55:11.2990874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ju106vg 2022-08-17T12:55:11.2991706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ju106vg/_remote_module_non_scriptable.py 2022-08-17T12:55:11.2993287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn90cudlp 2022-08-17T12:55:11.2996167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn90cudlp/_remote_module_non_scriptable.py 2022-08-17T12:55:11.3228580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmwyvs1k 2022-08-17T12:55:11.3229863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmwyvs1k/_remote_module_non_scriptable.py 2022-08-17T12:55:11.5560911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:11.5636278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:11.5798766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:11.5825614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:13.8414115Z STAGE:2022-08-17 12:55:13 22729:22729 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T12:55:13.8415154Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator()) 2022-08-17T12:55:14.8444280Z STAGE:2022-08-17 12:55:14 22729:22729 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T12:55:15.2678971Z ok (5.483s) 2022-08-17T12:55:15.2679221Z 2022-08-17T12:55:15.2679685Z ---------------------------------------------------------------------- 2022-08-17T12:55:15.2680106Z Ran 1 test in 5.483s 2022-08-17T12:55:15.2680325Z 2022-08-17T12:55:15.2680418Z OK 2022-08-17T12:55:15.2680558Z 2022-08-17T12:55:15.2680702Z Generating XML reports... 2022-08-17T12:55:15.2715675Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125509.xml 2022-08-17T12:55:16.5749377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:16.5750017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:16.5752027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:16.5752682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:16.7263454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprbxwubb 2022-08-17T12:55:16.7266439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprbxwubb/_remote_module_non_scriptable.py 2022-08-17T12:55:16.9893154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:16.9904001Z 2022-08-17T12:55:16.9904410Z Running tests... 2022-08-17T12:55:16.9904924Z ---------------------------------------------------------------------- 2022-08-17T12:55:17.2509739Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22952 2022-08-17T12:55:17.2537169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22953 2022-08-17T12:55:17.2565346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22954 2022-08-17T12:55:17.2597937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22955 2022-08-17T12:55:18.2746317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:18.2746955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:18.2750526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:18.2751176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:18.3030900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:18.3031523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:18.3034708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:18.3035349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:18.3183805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:18.3184468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:18.3187920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:18.3188573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:18.3426429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:18.3427142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:18.3429606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:18.3430284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:18.4335111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp623t5ax8 2022-08-17T12:55:18.4335934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp623t5ax8/_remote_module_non_scriptable.py 2022-08-17T12:55:18.4612578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mowv_d6 2022-08-17T12:55:18.4613342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mowv_d6/_remote_module_non_scriptable.py 2022-08-17T12:55:18.4783535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo_5zn71 2022-08-17T12:55:18.4784331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo_5zn71/_remote_module_non_scriptable.py 2022-08-17T12:55:18.5001409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanbsxwh2 2022-08-17T12:55:18.5002469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanbsxwh2/_remote_module_non_scriptable.py 2022-08-17T12:55:18.6928662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:18.7204824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:18.7379132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:18.7602529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:19.4657553Z ok (2.475s) 2022-08-17T12:55:19.4657821Z 2022-08-17T12:55:19.4658356Z ---------------------------------------------------------------------- 2022-08-17T12:55:19.4658708Z Ran 1 test in 2.475s 2022-08-17T12:55:19.4658830Z 2022-08-17T12:55:19.4658898Z OK 2022-08-17T12:55:19.4658993Z 2022-08-17T12:55:19.4659091Z Generating XML reports... 2022-08-17T12:55:19.4691967Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125516.xml 2022-08-17T12:55:20.7615179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:20.7615814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:20.7618725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:20.7619709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:20.9149527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovmapj6a 2022-08-17T12:55:20.9152524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovmapj6a/_remote_module_non_scriptable.py 2022-08-17T12:55:21.1848272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:21.1858820Z 2022-08-17T12:55:21.1858957Z Running tests... 2022-08-17T12:55:21.1859673Z ---------------------------------------------------------------------- 2022-08-17T12:55:21.4518145Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23175 2022-08-17T12:55:21.4546770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23176 2022-08-17T12:55:21.4575615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23177 2022-08-17T12:55:21.4606251Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23178 2022-08-17T12:55:22.4865351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:22.4866327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:22.4868958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:22.4869628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:22.5218973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:22.5219625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:22.5220405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:22.5221000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:22.5223859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:22.5224510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:22.5225422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:22.5226019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:22.5275282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:22.5275881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:22.5280355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:22.5280969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:22.6465913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_i_cy38 2022-08-17T12:55:22.6466938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_i_cy38/_remote_module_non_scriptable.py 2022-08-17T12:55:22.6819271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns1dw_yn 2022-08-17T12:55:22.6820037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm351zwfx 2022-08-17T12:55:22.6820729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns1dw_yn/_remote_module_non_scriptable.py 2022-08-17T12:55:22.6821465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm351zwfx/_remote_module_non_scriptable.py 2022-08-17T12:55:22.6884028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2em0bthy 2022-08-17T12:55:22.6885344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2em0bthy/_remote_module_non_scriptable.py 2022-08-17T12:55:22.9095873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:22.9439306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:22.9441919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:22.9543874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:23.5665142Z ok (2.380s) 2022-08-17T12:55:23.5665370Z 2022-08-17T12:55:23.5665700Z ---------------------------------------------------------------------- 2022-08-17T12:55:23.5665976Z Ran 1 test in 2.381s 2022-08-17T12:55:23.5666097Z 2022-08-17T12:55:23.5666161Z OK 2022-08-17T12:55:23.5666256Z 2022-08-17T12:55:23.5666337Z Generating XML reports... 2022-08-17T12:55:23.5700160Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125521.xml 2022-08-17T12:55:24.9027854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:24.9028233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:24.9030940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:24.9031329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:25.0563553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqw0t5ry 2022-08-17T12:55:25.0568448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqw0t5ry/_remote_module_non_scriptable.py 2022-08-17T12:55:25.3335132Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:25.3346351Z 2022-08-17T12:55:25.3346672Z Running tests... 2022-08-17T12:55:25.3347346Z ---------------------------------------------------------------------- 2022-08-17T12:55:25.5968779Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23398 2022-08-17T12:55:25.5997429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23399 2022-08-17T12:55:25.6027211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23400 2022-08-17T12:55:25.6058535Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23401 2022-08-17T12:55:26.6178016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:26.6178648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:26.6181462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:26.6182130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:26.6195931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:26.6196531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:26.6200280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:26.6200926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:26.6484063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:26.6484671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:26.6488190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:26.6488867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:26.6733221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:26.6733904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:26.6737285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:26.6737893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:26.7788265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjgr3r5i 2022-08-17T12:55:26.7789074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjgr3r5i/_remote_module_non_scriptable.py 2022-08-17T12:55:26.7789811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow4wf00j 2022-08-17T12:55:26.7792293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow4wf00j/_remote_module_non_scriptable.py 2022-08-17T12:55:26.8092345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfnij_ml 2022-08-17T12:55:26.8093299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfnij_ml/_remote_module_non_scriptable.py 2022-08-17T12:55:26.8322340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bk5370a 2022-08-17T12:55:26.8323817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bk5370a/_remote_module_non_scriptable.py 2022-08-17T12:55:27.0367948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:27.0399172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:27.0724757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:27.0943510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:27.8119542Z ok (2.477s) 2022-08-17T12:55:27.8119797Z 2022-08-17T12:55:27.8120358Z ---------------------------------------------------------------------- 2022-08-17T12:55:27.8120700Z Ran 1 test in 2.477s 2022-08-17T12:55:27.8120823Z 2022-08-17T12:55:27.8120873Z OK 2022-08-17T12:55:27.8120971Z 2022-08-17T12:55:27.8121065Z Generating XML reports... 2022-08-17T12:55:27.8157311Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125525.xml 2022-08-17T12:55:29.1029306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:29.1029695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:29.1032216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:29.1032593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:29.2512488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzngqjys 2022-08-17T12:55:29.2515379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzngqjys/_remote_module_non_scriptable.py 2022-08-17T12:55:29.5120022Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:29.5130558Z 2022-08-17T12:55:29.5130732Z Running tests... 2022-08-17T12:55:29.5131150Z ---------------------------------------------------------------------- 2022-08-17T12:55:29.7738628Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23621 2022-08-17T12:55:29.7767264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23622 2022-08-17T12:55:29.7796551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23623 2022-08-17T12:55:29.7828436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23624 2022-08-17T12:55:30.7903457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:30.7904497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:30.7907270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:30.7907932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:30.8157229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:30.8157956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:30.8161068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:30.8161721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:30.8162564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:30.8163255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:30.8164363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:30.8165301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:30.8190250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:30.8190900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:30.8195059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:30.8195729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:30.9489632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfcvi1ap 2022-08-17T12:55:30.9490493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfcvi1ap/_remote_module_non_scriptable.py 2022-08-17T12:55:30.9752800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyzllad5 2022-08-17T12:55:30.9753562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakrayhig 2022-08-17T12:55:30.9754599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyzllad5/_remote_module_non_scriptable.py 2022-08-17T12:55:30.9755543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakrayhig/_remote_module_non_scriptable.py 2022-08-17T12:55:30.9765815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2rjwjst 2022-08-17T12:55:30.9768272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2rjwjst/_remote_module_non_scriptable.py 2022-08-17T12:55:31.2109969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:31.2366489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:31.2370517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:31.2374841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:31.8886600Z ok (2.375s) 2022-08-17T12:55:31.8886854Z 2022-08-17T12:55:31.8887354Z ---------------------------------------------------------------------- 2022-08-17T12:55:31.8887632Z Ran 1 test in 2.375s 2022-08-17T12:55:31.8887755Z 2022-08-17T12:55:31.8888328Z OK 2022-08-17T12:55:31.8888494Z 2022-08-17T12:55:31.8888677Z Generating XML reports... 2022-08-17T12:55:31.8921045Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125529.xml 2022-08-17T12:55:33.1904211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:33.1904924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:33.1907087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:33.1907616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:33.3387940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nnyhqw7 2022-08-17T12:55:33.3391207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nnyhqw7/_remote_module_non_scriptable.py 2022-08-17T12:55:33.5989035Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:33.5998374Z 2022-08-17T12:55:33.5998517Z Running tests... 2022-08-17T12:55:33.5999158Z ---------------------------------------------------------------------- 2022-08-17T12:55:33.8571982Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23844 2022-08-17T12:55:33.8602321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23845 2022-08-17T12:55:33.8630523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23846 2022-08-17T12:55:33.8661819Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23847 2022-08-17T12:55:34.9200462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:34.9201133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:34.9202600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:34.9203211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:34.9203993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:34.9204637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:34.9209575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:34.9210247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:34.9515632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:34.9516309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:34.9517609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:34.9518262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:34.9821347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:34.9821975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:34.9825113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:34.9825711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:35.0790094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_ufao7d 2022-08-17T12:55:35.0791293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_ufao7d/_remote_module_non_scriptable.py 2022-08-17T12:55:35.0800073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffp9vedl 2022-08-17T12:55:35.0802203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffp9vedl/_remote_module_non_scriptable.py 2022-08-17T12:55:35.1087507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n4cfyh4 2022-08-17T12:55:35.1088336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n4cfyh4/_remote_module_non_scriptable.py 2022-08-17T12:55:35.1387550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz9w2exu 2022-08-17T12:55:35.1388695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz9w2exu/_remote_module_non_scriptable.py 2022-08-17T12:55:35.3395400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:35.3410964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:35.3685107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:35.4034418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:39.1771765Z ok (5.577s) 2022-08-17T12:55:39.1772033Z 2022-08-17T12:55:39.1772577Z ---------------------------------------------------------------------- 2022-08-17T12:55:39.1772941Z Ran 1 test in 5.577s 2022-08-17T12:55:39.1773064Z 2022-08-17T12:55:39.1773126Z OK 2022-08-17T12:55:39.1773239Z 2022-08-17T12:55:39.1773337Z Generating XML reports... 2022-08-17T12:55:39.1805550Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125533.xml 2022-08-17T12:55:40.4917213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:40.4917621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:40.4920143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:40.4920524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:40.6414079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt2o7rje 2022-08-17T12:55:40.6417562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt2o7rje/_remote_module_non_scriptable.py 2022-08-17T12:55:40.9028630Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:40.9039622Z 2022-08-17T12:55:40.9039883Z Running tests... 2022-08-17T12:55:40.9040548Z ---------------------------------------------------------------------- 2022-08-17T12:55:41.1647040Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24067 2022-08-17T12:55:41.1676797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24068 2022-08-17T12:55:41.1706708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24069 2022-08-17T12:55:41.1740130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24070 2022-08-17T12:55:42.2520910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:42.2521525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:42.2524384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:42.2525035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:42.2796701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:42.2797290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:42.2800032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:42.2800693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:42.2941582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:42.2942308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:42.2943147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:42.2944110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:42.2945300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:42.2945999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:42.2947333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:42.2948007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:42.4105825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmc_zy1t 2022-08-17T12:55:42.4106577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmc_zy1t/_remote_module_non_scriptable.py 2022-08-17T12:55:42.4384202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvgd0aa2 2022-08-17T12:55:42.4385440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvgd0aa2/_remote_module_non_scriptable.py 2022-08-17T12:55:42.4535962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt35xf3y6 2022-08-17T12:55:42.4537204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt35xf3y6/_remote_module_non_scriptable.py 2022-08-17T12:55:42.4541949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv86a1bsq 2022-08-17T12:55:42.4544418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv86a1bsq/_remote_module_non_scriptable.py 2022-08-17T12:55:42.6728612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:42.6996484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:42.7159098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:42.7185338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:46.3847712Z ok (5.480s) 2022-08-17T12:55:46.3847951Z 2022-08-17T12:55:46.3848400Z ---------------------------------------------------------------------- 2022-08-17T12:55:46.3848684Z Ran 1 test in 5.481s 2022-08-17T12:55:46.3848806Z 2022-08-17T12:55:46.3848885Z OK 2022-08-17T12:55:46.3848996Z 2022-08-17T12:55:46.3849135Z Generating XML reports... 2022-08-17T12:55:46.3881836Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125540.xml 2022-08-17T12:55:47.6657732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:47.6658159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:47.6660750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:47.6661305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:47.8184288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstyjtclq 2022-08-17T12:55:47.8188196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstyjtclq/_remote_module_non_scriptable.py 2022-08-17T12:55:48.0840263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:48.0851384Z 2022-08-17T12:55:48.0851523Z Running tests... 2022-08-17T12:55:48.0852160Z ---------------------------------------------------------------------- 2022-08-17T12:55:48.3557576Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24290 2022-08-17T12:55:48.3588987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24291 2022-08-17T12:55:48.3618667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24292 2022-08-17T12:55:48.3650805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24293 2022-08-17T12:55:49.4712470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:49.4713169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:49.4715805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:49.4716465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:49.4753037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:49.4753639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:49.4756798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:49.4757466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:49.5035953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:49.5036609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:49.5040063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:49.5040739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:49.5547147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:49.5547791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:49.5551856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:49.5552510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:49.6319129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbunlk5j 2022-08-17T12:55:49.6320022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbunlk5j/_remote_module_non_scriptable.py 2022-08-17T12:55:49.6353536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmaaaa96 2022-08-17T12:55:49.6355524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmaaaa96/_remote_module_non_scriptable.py 2022-08-17T12:55:49.6606706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8k8jkuq 2022-08-17T12:55:49.6608287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8k8jkuq/_remote_module_non_scriptable.py 2022-08-17T12:55:49.7117162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbatg9rqk 2022-08-17T12:55:49.7118299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbatg9rqk/_remote_module_non_scriptable.py 2022-08-17T12:55:49.8929894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:49.8969968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:49.9189159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:49.9751654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:50.5708763Z ok (2.485s) 2022-08-17T12:55:50.5709012Z 2022-08-17T12:55:50.5709470Z ---------------------------------------------------------------------- 2022-08-17T12:55:50.5709886Z Ran 1 test in 2.486s 2022-08-17T12:55:50.5710059Z 2022-08-17T12:55:50.5710165Z OK 2022-08-17T12:55:50.5712343Z 2022-08-17T12:55:50.5712683Z Generating XML reports... 2022-08-17T12:55:50.5745208Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125548.xml 2022-08-17T12:55:51.8778321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:51.8778701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:51.8780755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:51.8781132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:52.0301589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3qfkl8g 2022-08-17T12:55:52.0304941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3qfkl8g/_remote_module_non_scriptable.py 2022-08-17T12:55:52.2946167Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:52.2956799Z 2022-08-17T12:55:52.2956917Z Running tests... 2022-08-17T12:55:52.2957982Z ---------------------------------------------------------------------- 2022-08-17T12:55:52.5591445Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24513 2022-08-17T12:55:52.5620496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24514 2022-08-17T12:55:52.5651675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24515 2022-08-17T12:55:52.5692764Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24516 2022-08-17T12:55:53.6316766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:53.6317396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:53.6318959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:53.6319579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:53.6621287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:53.6621952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:53.6625345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:53.6625997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:53.6680567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:53.6681261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:53.6685252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:53.6686066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:53.6720271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:53.6720921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:53.6724483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:53.6725094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:53.7902494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvqnfh69 2022-08-17T12:55:53.7903594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvqnfh69/_remote_module_non_scriptable.py 2022-08-17T12:55:53.8237169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy61k6be 2022-08-17T12:55:53.8237950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy61k6be/_remote_module_non_scriptable.py 2022-08-17T12:55:53.8282494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kmbvudc 2022-08-17T12:55:53.8284036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kmbvudc/_remote_module_non_scriptable.py 2022-08-17T12:55:53.8325567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlwag3x1 2022-08-17T12:55:53.8327507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlwag3x1/_remote_module_non_scriptable.py 2022-08-17T12:55:54.0542014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:54.0874244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:54.0907134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:54.0958279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:54.7750473Z ok (2.479s) 2022-08-17T12:55:54.7750711Z 2022-08-17T12:55:54.7751261Z ---------------------------------------------------------------------- 2022-08-17T12:55:54.7751611Z Ran 1 test in 2.479s 2022-08-17T12:55:54.7751737Z 2022-08-17T12:55:54.7751789Z OK 2022-08-17T12:55:54.7751884Z 2022-08-17T12:55:54.7751978Z Generating XML reports... 2022-08-17T12:55:54.7788397Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125552.xml 2022-08-17T12:55:56.0886910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:56.0887300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:56.0890204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:56.0890702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:56.2441580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_z0m5rr 2022-08-17T12:55:56.2445259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_z0m5rr/_remote_module_non_scriptable.py 2022-08-17T12:55:56.5177566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:55:56.5188287Z 2022-08-17T12:55:56.5188420Z Running tests... 2022-08-17T12:55:56.5189178Z ---------------------------------------------------------------------- 2022-08-17T12:55:56.7855272Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24736 2022-08-17T12:55:56.7884754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24737 2022-08-17T12:55:56.7913414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24738 2022-08-17T12:55:56.7945293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24739 2022-08-17T12:55:57.7978115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:57.7978801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:57.7980816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:57.7981470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:57.8187230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:57.8187847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:57.8188670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:57.8189334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:57.8190082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:57.8191002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:57.8192171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:57.8192759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:57.8304141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:55:57.8304916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:55:57.8308575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:55:57.8309249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:55:57.9563405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlzxv2hz 2022-08-17T12:55:57.9564192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlzxv2hz/_remote_module_non_scriptable.py 2022-08-17T12:55:57.9766351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagl7kssd 2022-08-17T12:55:57.9767989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagl7kssd/_remote_module_non_scriptable.py 2022-08-17T12:55:57.9788371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_542kb6 2022-08-17T12:55:57.9789684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_542kb6/_remote_module_non_scriptable.py 2022-08-17T12:55:57.9893452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkupve9w 2022-08-17T12:55:57.9895054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkupve9w/_remote_module_non_scriptable.py 2022-08-17T12:55:58.2163735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:55:58.2352141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:55:58.2386665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:55:58.2485304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:55:59.0005499Z ok (2.481s) 2022-08-17T12:55:59.0005762Z 2022-08-17T12:55:59.0006447Z ---------------------------------------------------------------------- 2022-08-17T12:55:59.0006735Z Ran 1 test in 2.482s 2022-08-17T12:55:59.0006859Z 2022-08-17T12:55:59.0006910Z OK 2022-08-17T12:55:59.0007006Z 2022-08-17T12:55:59.0007100Z Generating XML reports... 2022-08-17T12:55:59.0040010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125556.xml 2022-08-17T12:56:00.3231789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:00.3232197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:00.3234228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:00.3234607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:00.4749681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdid4ogcg 2022-08-17T12:56:00.4753093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdid4ogcg/_remote_module_non_scriptable.py 2022-08-17T12:56:00.7412310Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:00.7423253Z 2022-08-17T12:56:00.7423476Z Running tests... 2022-08-17T12:56:00.7423894Z ---------------------------------------------------------------------- 2022-08-17T12:56:01.0075289Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24959 2022-08-17T12:56:01.0105951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24960 2022-08-17T12:56:01.0135737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24961 2022-08-17T12:56:01.0169183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24962 2022-08-17T12:56:02.1088446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:02.1089074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:02.1091681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:02.1092337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:02.1260278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:02.1260926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:02.1264450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:02.1265199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:02.1289512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:02.1290118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:02.1294896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:02.1295566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:02.1422777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:02.1423389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:02.1427001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:02.1427698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:02.2688750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvtdu6pl 2022-08-17T12:56:02.2690023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvtdu6pl/_remote_module_non_scriptable.py 2022-08-17T12:56:02.2881758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx90y97n 2022-08-17T12:56:02.2882575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx90y97n/_remote_module_non_scriptable.py 2022-08-17T12:56:02.2898954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqheypmm 2022-08-17T12:56:02.2900811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqheypmm/_remote_module_non_scriptable.py 2022-08-17T12:56:02.2999806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmm3vjj9 2022-08-17T12:56:02.3001440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmm3vjj9/_remote_module_non_scriptable.py 2022-08-17T12:56:02.5301651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:02.5490326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:02.5500599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:02.5558904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:03.2228667Z ok (2.480s) 2022-08-17T12:56:03.2228885Z 2022-08-17T12:56:03.2229232Z ---------------------------------------------------------------------- 2022-08-17T12:56:03.2229497Z Ran 1 test in 2.480s 2022-08-17T12:56:03.2229604Z 2022-08-17T12:56:03.2229667Z OK 2022-08-17T12:56:03.2229764Z 2022-08-17T12:56:03.2229860Z Generating XML reports... 2022-08-17T12:56:03.2263201Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125600.xml 2022-08-17T12:56:04.5207006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:04.5207427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:04.5209768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:04.5210188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:04.6737861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac7bdjo6 2022-08-17T12:56:04.6741173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac7bdjo6/_remote_module_non_scriptable.py 2022-08-17T12:56:04.9411239Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:04.9422347Z 2022-08-17T12:56:04.9422801Z Running tests... 2022-08-17T12:56:04.9423274Z ---------------------------------------------------------------------- 2022-08-17T12:56:05.2130913Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25182 2022-08-17T12:56:05.2158093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25183 2022-08-17T12:56:05.2187790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25184 2022-08-17T12:56:05.2220656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25185 2022-08-17T12:56:06.3074992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:06.3075627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:06.3078743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:06.3079395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:06.3262865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:06.3263494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:06.3266617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:06.3267278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:06.3446401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:06.3447034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:06.3449630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:06.3450299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:06.3562819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:06.3563476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:06.3566992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:06.3567624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:06.4676934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegloacmh 2022-08-17T12:56:06.4677743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegloacmh/_remote_module_non_scriptable.py 2022-08-17T12:56:06.4856571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgspl0bty 2022-08-17T12:56:06.4857419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgspl0bty/_remote_module_non_scriptable.py 2022-08-17T12:56:06.5041223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86l78d4h 2022-08-17T12:56:06.5042059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86l78d4h/_remote_module_non_scriptable.py 2022-08-17T12:56:06.5151102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8cdm842 2022-08-17T12:56:06.5151805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8cdm842/_remote_module_non_scriptable.py 2022-08-17T12:56:06.7289672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:06.7460530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:06.7659506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:06.7780218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:07.4282391Z ok (2.486s) 2022-08-17T12:56:07.4282616Z 2022-08-17T12:56:07.4283096Z ---------------------------------------------------------------------- 2022-08-17T12:56:07.4283503Z Ran 1 test in 2.486s 2022-08-17T12:56:07.4283705Z 2022-08-17T12:56:07.4283810Z OK 2022-08-17T12:56:07.4283955Z 2022-08-17T12:56:07.4284374Z Generating XML reports... 2022-08-17T12:56:07.4317585Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125604.xml 2022-08-17T12:56:08.7418430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:08.7418842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:08.7420722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:08.7421102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:08.8941880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7sskkmw 2022-08-17T12:56:08.8944899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7sskkmw/_remote_module_non_scriptable.py 2022-08-17T12:56:09.1564157Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:09.1575117Z 2022-08-17T12:56:09.1575214Z Running tests... 2022-08-17T12:56:09.1576267Z ---------------------------------------------------------------------- 2022-08-17T12:56:09.4220161Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25405 2022-08-17T12:56:09.4249099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25406 2022-08-17T12:56:09.4277813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25407 2022-08-17T12:56:09.4311697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25408 2022-08-17T12:56:10.4442077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:10.4442664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:10.4445359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:10.4446163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:10.4646766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:10.4647398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:10.4650211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:10.4650870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:10.4720885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:10.4721559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:10.4726395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:10.4727070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:10.5058863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:10.5059377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:10.5063105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:10.5063792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:10.6027064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9r8q3q2 2022-08-17T12:56:10.6027855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9r8q3q2/_remote_module_non_scriptable.py 2022-08-17T12:56:10.6239122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0ri_yc7 2022-08-17T12:56:10.6240166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0ri_yc7/_remote_module_non_scriptable.py 2022-08-17T12:56:10.6305432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmt8pioi 2022-08-17T12:56:10.6307030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmt8pioi/_remote_module_non_scriptable.py 2022-08-17T12:56:10.6636448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx802m4o1 2022-08-17T12:56:10.6637829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx802m4o1/_remote_module_non_scriptable.py 2022-08-17T12:56:10.8623302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:10.8846090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:10.8894348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:10.9259555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:11.5369302Z ok (2.379s) 2022-08-17T12:56:11.5369529Z 2022-08-17T12:56:11.5369869Z ---------------------------------------------------------------------- 2022-08-17T12:56:11.5370188Z Ran 1 test in 2.379s 2022-08-17T12:56:11.5370355Z 2022-08-17T12:56:11.5370422Z OK 2022-08-17T12:56:11.5370519Z 2022-08-17T12:56:11.5370612Z Generating XML reports... 2022-08-17T12:56:11.5404210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125609.xml 2022-08-17T12:56:12.8421543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:12.8421990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:12.8424609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:12.8425048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:12.9935838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp546mtj97 2022-08-17T12:56:12.9939062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp546mtj97/_remote_module_non_scriptable.py 2022-08-17T12:56:13.2555012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:13.2565528Z 2022-08-17T12:56:13.2565642Z Running tests... 2022-08-17T12:56:13.2566286Z ---------------------------------------------------------------------- 2022-08-17T12:56:13.5201371Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25628 2022-08-17T12:56:13.5230861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25629 2022-08-17T12:56:13.5260887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25630 2022-08-17T12:56:13.5292461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25631 2022-08-17T12:56:14.5213609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:14.5214291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:14.5217082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:14.5217783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:14.5371377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:14.5372029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:14.5374447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:14.5375379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:14.5537335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:14.5538038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:14.5541709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:14.5542395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:14.5580852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:14.5581495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:14.5585176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:14.5585859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:14.6808473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvl97aeg 2022-08-17T12:56:14.6809729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvl97aeg/_remote_module_non_scriptable.py 2022-08-17T12:56:14.6960567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl175n0hz 2022-08-17T12:56:14.6961852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl175n0hz/_remote_module_non_scriptable.py 2022-08-17T12:56:14.7143003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0glnftbo 2022-08-17T12:56:14.7144101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0glnftbo/_remote_module_non_scriptable.py 2022-08-17T12:56:14.7169221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp944brspw 2022-08-17T12:56:14.7171273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp944brspw/_remote_module_non_scriptable.py 2022-08-17T12:56:14.9436952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:14.9549201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:14.9762943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:14.9765781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:15.6351554Z ok (2.378s) 2022-08-17T12:56:15.6351827Z 2022-08-17T12:56:15.6352283Z ---------------------------------------------------------------------- 2022-08-17T12:56:15.6352825Z Ran 1 test in 2.378s 2022-08-17T12:56:15.6352933Z 2022-08-17T12:56:15.6352998Z OK 2022-08-17T12:56:15.6353096Z 2022-08-17T12:56:15.6353192Z Generating XML reports... 2022-08-17T12:56:15.6386884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125613.xml 2022-08-17T12:56:16.9348052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:16.9348453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:16.9350637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:16.9351145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:17.0853518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps00wf90i 2022-08-17T12:56:17.0856465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps00wf90i/_remote_module_non_scriptable.py 2022-08-17T12:56:17.3486608Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:17.3496664Z 2022-08-17T12:56:17.3496784Z Running tests... 2022-08-17T12:56:17.3497677Z ---------------------------------------------------------------------- 2022-08-17T12:56:17.6182617Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25851 2022-08-17T12:56:17.6210259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25852 2022-08-17T12:56:17.6239051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25853 2022-08-17T12:56:17.6270497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25854 2022-08-17T12:56:18.6715240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:18.6715922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:18.6719321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:18.6719990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:18.6759439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:18.6760110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:18.6761176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:18.6761781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:18.6763370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:18.6764045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:18.6767293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:18.6767964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:18.6779278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:18.6779902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:18.6783598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:18.6784257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:18.8319259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje8qce9g 2022-08-17T12:56:18.8320206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje8qce9g/_remote_module_non_scriptable.py 2022-08-17T12:56:18.8362397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy97pa764 2022-08-17T12:56:18.8363268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqk42itn 2022-08-17T12:56:18.8363942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy97pa764/_remote_module_non_scriptable.py 2022-08-17T12:56:18.8366800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqk42itn/_remote_module_non_scriptable.py 2022-08-17T12:56:18.8373493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8gdu0y0 2022-08-17T12:56:18.8375631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8gdu0y0/_remote_module_non_scriptable.py 2022-08-17T12:56:19.0935120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:19.0954797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:19.0989863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:19.1027401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:19.7330016Z ok (2.383s) 2022-08-17T12:56:19.7330256Z 2022-08-17T12:56:19.7331106Z ---------------------------------------------------------------------- 2022-08-17T12:56:19.7331381Z Ran 1 test in 2.383s 2022-08-17T12:56:19.7331502Z 2022-08-17T12:56:19.7331552Z OK 2022-08-17T12:56:19.7331649Z 2022-08-17T12:56:19.7331746Z Generating XML reports... 2022-08-17T12:56:19.7364075Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125617.xml 2022-08-17T12:56:21.0475290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:21.0475716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:21.0478587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:21.0478990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:21.1973681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9228nwh 2022-08-17T12:56:21.1977036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9228nwh/_remote_module_non_scriptable.py 2022-08-17T12:56:21.4591246Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:21.4602944Z 2022-08-17T12:56:21.4603049Z Running tests... 2022-08-17T12:56:21.4603546Z ---------------------------------------------------------------------- 2022-08-17T12:56:21.7206515Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26074 2022-08-17T12:56:21.7242786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26075 2022-08-17T12:56:21.7287727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26076 2022-08-17T12:56:21.7318469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26077 2022-08-17T12:56:22.7309397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:22.7310060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:22.7312872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:22.7313543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:22.7611916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:22.7612608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:22.7616297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:22.7617276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:22.7633228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:22.7633828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:22.7638088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:22.7638688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:22.7809590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:22.7810298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:22.7811870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:22.7812574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:22.8967562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1yw9zm_ 2022-08-17T12:56:22.8969160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1yw9zm_/_remote_module_non_scriptable.py 2022-08-17T12:56:22.9228512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zxswl6b 2022-08-17T12:56:22.9229349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zxswl6b/_remote_module_non_scriptable.py 2022-08-17T12:56:22.9243355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6gxwt0_ 2022-08-17T12:56:22.9244914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6gxwt0_/_remote_module_non_scriptable.py 2022-08-17T12:56:22.9457668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrj_npsh 2022-08-17T12:56:22.9458719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrj_npsh/_remote_module_non_scriptable.py 2022-08-17T12:56:23.1561570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:23.1815243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:23.1834862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:23.2024828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:25.8407822Z ok (4.380s) 2022-08-17T12:56:25.8408113Z 2022-08-17T12:56:25.8408537Z ---------------------------------------------------------------------- 2022-08-17T12:56:25.8408791Z Ran 1 test in 4.380s 2022-08-17T12:56:25.8408924Z 2022-08-17T12:56:25.8408989Z OK 2022-08-17T12:56:25.8409120Z 2022-08-17T12:56:25.8409243Z Generating XML reports... 2022-08-17T12:56:25.8442791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125621.xml 2022-08-17T12:56:27.1528520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:27.1529250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:27.1532248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:27.1532849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:27.3056514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfq0tsz54 2022-08-17T12:56:27.3060436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfq0tsz54/_remote_module_non_scriptable.py 2022-08-17T12:56:27.5702115Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:27.5713982Z 2022-08-17T12:56:27.5714569Z Running tests... 2022-08-17T12:56:27.5715003Z ---------------------------------------------------------------------- 2022-08-17T12:56:27.8366811Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26297 2022-08-17T12:56:27.8394574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26298 2022-08-17T12:56:27.8423008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26299 2022-08-17T12:56:27.8455801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26300 2022-08-17T12:56:28.9234449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:28.9235158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:28.9237941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:28.9238565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:28.9350344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:28.9351365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:28.9355074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:28.9355725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:28.9655495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:28.9656241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:28.9658984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:28.9659699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:28.9934601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:28.9935299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:28.9937334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:28.9937909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:29.0819955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mkjsxv8 2022-08-17T12:56:29.0821021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mkjsxv8/_remote_module_non_scriptable.py 2022-08-17T12:56:29.0923143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbosc4mm 2022-08-17T12:56:29.0924461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbosc4mm/_remote_module_non_scriptable.py 2022-08-17T12:56:29.1242586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9re_byxa 2022-08-17T12:56:29.1243405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9re_byxa/_remote_module_non_scriptable.py 2022-08-17T12:56:29.1491259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha_hskc4 2022-08-17T12:56:29.1492586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha_hskc4/_remote_module_non_scriptable.py 2022-08-17T12:56:29.3485505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:29.3723581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:29.3874939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:29.4083114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:32.0544786Z ok (4.483s) 2022-08-17T12:56:32.0545056Z 2022-08-17T12:56:32.0545616Z ---------------------------------------------------------------------- 2022-08-17T12:56:32.0546064Z Ran 1 test in 4.483s 2022-08-17T12:56:32.0546172Z 2022-08-17T12:56:32.0546238Z OK 2022-08-17T12:56:32.0546339Z 2022-08-17T12:56:32.0546457Z Generating XML reports... 2022-08-17T12:56:32.0578814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125627.xml 2022-08-17T12:56:33.3388911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:33.3389300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:33.3391682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:33.3392061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:33.4908361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pk7amcw 2022-08-17T12:56:33.4911204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pk7amcw/_remote_module_non_scriptable.py 2022-08-17T12:56:33.7514169Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:33.7524745Z 2022-08-17T12:56:33.7524900Z Running tests... 2022-08-17T12:56:33.7525330Z ---------------------------------------------------------------------- 2022-08-17T12:56:34.0119943Z test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26520 2022-08-17T12:56:34.0147901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26521 2022-08-17T12:56:34.0175601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26522 2022-08-17T12:56:34.0206366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26523 2022-08-17T12:56:35.0609770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:35.0610435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:35.0612940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:35.0613538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:35.0650357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:35.0651015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:35.0655587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:35.0657125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:35.0713974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:35.0714634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:35.0718173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:35.0718843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:35.0755297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:35.0755931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:35.0760176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:35.0760836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:35.2207775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyjvsu3p 2022-08-17T12:56:35.2208952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyjvsu3p/_remote_module_non_scriptable.py 2022-08-17T12:56:35.2257337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj76i2ot1 2022-08-17T12:56:35.2258878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj76i2ot1/_remote_module_non_scriptable.py 2022-08-17T12:56:35.2303849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg99l0d10 2022-08-17T12:56:35.2305531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg99l0d10/_remote_module_non_scriptable.py 2022-08-17T12:56:35.2357726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrn6r2kg 2022-08-17T12:56:35.2359724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrn6r2kg/_remote_module_non_scriptable.py 2022-08-17T12:56:35.4815290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:35.4916236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:35.4955919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:35.4994597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:36.1265130Z ok (2.374s) 2022-08-17T12:56:36.1265393Z 2022-08-17T12:56:36.1265923Z ---------------------------------------------------------------------- 2022-08-17T12:56:36.1266321Z Ran 1 test in 2.374s 2022-08-17T12:56:36.1266446Z 2022-08-17T12:56:36.1266513Z OK 2022-08-17T12:56:36.1266611Z 2022-08-17T12:56:36.1266707Z Generating XML reports... 2022-08-17T12:56:36.1299028Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125633.xml 2022-08-17T12:56:37.4181618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:37.4182058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:37.4184353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:37.4184808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:37.5705188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgac98edh 2022-08-17T12:56:37.5708813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgac98edh/_remote_module_non_scriptable.py 2022-08-17T12:56:37.8349026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:37.8361010Z 2022-08-17T12:56:37.8361149Z Running tests... 2022-08-17T12:56:37.8361644Z ---------------------------------------------------------------------- 2022-08-17T12:56:38.0995947Z test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26743 2022-08-17T12:56:38.1025615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26744 2022-08-17T12:56:38.1053969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26745 2022-08-17T12:56:38.1084744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26746 2022-08-17T12:56:39.0817285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:39.0817957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:39.0821022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:39.0821724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:39.1041570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:39.1042565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:39.1046566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:39.1047250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:39.1076154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:39.1076795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:39.1080547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:39.1081224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:39.1354907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:39.1355572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:39.1358395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:39.1358992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:39.2397418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfo06yap 2022-08-17T12:56:39.2398302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfo06yap/_remote_module_non_scriptable.py 2022-08-17T12:56:39.2628501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8kgbww6 2022-08-17T12:56:39.2629359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8kgbww6/_remote_module_non_scriptable.py 2022-08-17T12:56:39.2650470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1jqmzw_ 2022-08-17T12:56:39.2652582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1jqmzw_/_remote_module_non_scriptable.py 2022-08-17T12:56:39.2925488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd13ywf4 2022-08-17T12:56:39.2926882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd13ywf4/_remote_module_non_scriptable.py 2022-08-17T12:56:39.5007566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:39.5212448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:39.5214961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:39.5534241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:40.2143365Z ok (2.378s) 2022-08-17T12:56:40.2143619Z 2022-08-17T12:56:40.2144097Z ---------------------------------------------------------------------- 2022-08-17T12:56:40.2144589Z Ran 1 test in 2.378s 2022-08-17T12:56:40.2144781Z 2022-08-17T12:56:40.2144873Z OK 2022-08-17T12:56:40.2145016Z 2022-08-17T12:56:40.2145155Z Generating XML reports... 2022-08-17T12:56:40.2179114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125637.xml 2022-08-17T12:56:41.5239112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:41.5239533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:41.5242273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:41.5242684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:41.6792807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhyjqqgh 2022-08-17T12:56:41.6795819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhyjqqgh/_remote_module_non_scriptable.py 2022-08-17T12:56:41.9456863Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:41.9467317Z 2022-08-17T12:56:41.9467478Z Running tests... 2022-08-17T12:56:41.9468076Z ---------------------------------------------------------------------- 2022-08-17T12:56:42.2086050Z test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26966 2022-08-17T12:56:42.2114021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26967 2022-08-17T12:56:42.2142215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26968 2022-08-17T12:56:42.2172529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26969 2022-08-17T12:56:43.1977085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:43.1977729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:43.1980767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:43.1981442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:43.2189656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:43.2190321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:43.2191132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:43.2191744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:43.2193073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:43.2193683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:43.2194497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:43.2195141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:43.2724367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:43.2725144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:43.2727499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:43.2728247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:43.3584581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7aomlpb 2022-08-17T12:56:43.3585464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7aomlpb/_remote_module_non_scriptable.py 2022-08-17T12:56:43.3796577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0jmun7i 2022-08-17T12:56:43.3797347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyq1nrl3 2022-08-17T12:56:43.3798075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0jmun7i/_remote_module_non_scriptable.py 2022-08-17T12:56:43.3798810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyq1nrl3/_remote_module_non_scriptable.py 2022-08-17T12:56:43.4293481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdttmzgi 2022-08-17T12:56:43.4294347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdttmzgi/_remote_module_non_scriptable.py 2022-08-17T12:56:43.6220320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:43.6434848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:43.6449187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:43.6918017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:44.3230661Z ok (2.376s) 2022-08-17T12:56:44.3230834Z 2022-08-17T12:56:44.3231173Z ---------------------------------------------------------------------- 2022-08-17T12:56:44.3231424Z Ran 1 test in 2.376s 2022-08-17T12:56:44.3231565Z 2022-08-17T12:56:44.3231632Z OK 2022-08-17T12:56:44.3231728Z 2022-08-17T12:56:44.3231826Z Generating XML reports... 2022-08-17T12:56:44.3267831Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125641.xml 2022-08-17T12:56:45.6164895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:45.6165301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:45.6168382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:45.6168778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:45.7691129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfm67o11 2022-08-17T12:56:45.7694804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfm67o11/_remote_module_non_scriptable.py 2022-08-17T12:56:46.0333139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:46.0342956Z 2022-08-17T12:56:46.0343225Z Running tests... 2022-08-17T12:56:46.0343771Z ---------------------------------------------------------------------- 2022-08-17T12:56:46.2983570Z test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27189 2022-08-17T12:56:46.3011104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27190 2022-08-17T12:56:46.3040993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27191 2022-08-17T12:56:46.3073164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27192 2022-08-17T12:56:47.3523085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:47.3523801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:47.3527093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:47.3527810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:47.3535731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:47.3536299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:47.3540463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:47.3541181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:47.3624162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:47.3624888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:47.3628270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:47.3628891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:47.3760670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:47.3761353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:47.3765763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:47.3766703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:47.5117549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45mms9sl 2022-08-17T12:56:47.5118375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45mms9sl/_remote_module_non_scriptable.py 2022-08-17T12:56:47.5135966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0phzi8n 2022-08-17T12:56:47.5137938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0phzi8n/_remote_module_non_scriptable.py 2022-08-17T12:56:47.5199374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8v6gcxn 2022-08-17T12:56:47.5201784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8v6gcxn/_remote_module_non_scriptable.py 2022-08-17T12:56:47.5338237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdeqwc48j 2022-08-17T12:56:47.5339224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdeqwc48j/_remote_module_non_scriptable.py 2022-08-17T12:56:47.7711672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:47.7763215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:47.7766111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:47.7956546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:48.4132929Z ok (2.379s) 2022-08-17T12:56:48.4133157Z 2022-08-17T12:56:48.4133640Z ---------------------------------------------------------------------- 2022-08-17T12:56:48.4134050Z Ran 1 test in 2.379s 2022-08-17T12:56:48.4134246Z 2022-08-17T12:56:48.4134342Z OK 2022-08-17T12:56:48.4134491Z 2022-08-17T12:56:48.4134624Z Generating XML reports... 2022-08-17T12:56:48.4168499Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125646.xml 2022-08-17T12:56:49.7191708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:49.7192099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:49.7193863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:49.7194318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:49.8701011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vagxgi_ 2022-08-17T12:56:49.8704259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vagxgi_/_remote_module_non_scriptable.py 2022-08-17T12:56:50.1342599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:50.1353249Z 2022-08-17T12:56:50.1353333Z Running tests... 2022-08-17T12:56:50.1354103Z ---------------------------------------------------------------------- 2022-08-17T12:56:50.3971853Z test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27412 2022-08-17T12:56:50.4002216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27413 2022-08-17T12:56:50.4031919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27414 2022-08-17T12:56:50.4063413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27415 2022-08-17T12:56:51.4903774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:51.4904568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:51.4907363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:51.4907980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:51.5164951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:51.5165606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:51.5166916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:51.5167829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:51.5169136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:51.5169826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:51.5172726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:51.5173311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:51.5394402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:51.5395049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:51.5397865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:51.5398807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:51.6489264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tuazew4 2022-08-17T12:56:51.6490755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tuazew4/_remote_module_non_scriptable.py 2022-08-17T12:56:51.6764620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi96peejc 2022-08-17T12:56:51.6765375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwavvpah_ 2022-08-17T12:56:51.6766243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi96peejc/_remote_module_non_scriptable.py 2022-08-17T12:56:51.6766910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwavvpah_/_remote_module_non_scriptable.py 2022-08-17T12:56:51.6986995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0v78aig8 2022-08-17T12:56:51.6987978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0v78aig8/_remote_module_non_scriptable.py 2022-08-17T12:56:51.9068088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:51.9356059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:51.9399039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:51.9610755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:52.6122499Z ok (2.477s) 2022-08-17T12:56:52.6122698Z 2022-08-17T12:56:52.6123155Z ---------------------------------------------------------------------- 2022-08-17T12:56:52.6123438Z Ran 1 test in 2.477s 2022-08-17T12:56:52.6123548Z 2022-08-17T12:56:52.6123617Z OK 2022-08-17T12:56:52.6123714Z 2022-08-17T12:56:52.6123809Z Generating XML reports... 2022-08-17T12:56:52.6158066Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125650.xml 2022-08-17T12:56:53.9079128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:53.9079830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:53.9083248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:53.9083646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:54.0597329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22cmdqrk 2022-08-17T12:56:54.0600317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22cmdqrk/_remote_module_non_scriptable.py 2022-08-17T12:56:54.3269840Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:54.3282596Z 2022-08-17T12:56:54.3282698Z Running tests... 2022-08-17T12:56:54.3283628Z ---------------------------------------------------------------------- 2022-08-17T12:56:54.5925677Z test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27635 2022-08-17T12:56:54.5953909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27636 2022-08-17T12:56:54.5983221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27637 2022-08-17T12:56:54.6014424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27638 2022-08-17T12:56:55.6430594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:55.6431295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:55.6433050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:55.6434004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:55.6764704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:55.6765408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:55.6768099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:55.6768753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:55.7279935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:55.7280572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:55.7283924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:55.7284592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:55.7444540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:55.7445176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:55.7448482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:55.7449099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:55.7993376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4si0dor8 2022-08-17T12:56:55.7994197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4si0dor8/_remote_module_non_scriptable.py 2022-08-17T12:56:55.8345499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsfk1a68 2022-08-17T12:56:55.8346278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsfk1a68/_remote_module_non_scriptable.py 2022-08-17T12:56:55.8876342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp746r_sp5 2022-08-17T12:56:55.8877210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp746r_sp5/_remote_module_non_scriptable.py 2022-08-17T12:56:55.9040824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9yo01nv 2022-08-17T12:56:55.9041787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9yo01nv/_remote_module_non_scriptable.py 2022-08-17T12:56:56.0573830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:56:56.0958172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:56:56.1508854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:56:56.1668198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:56:56.3754692Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:56:56.3755681Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:56:56.3756304Z Traceback (most recent call last): 2022-08-17T12:56:56.3757241Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:56:56.3758024Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:56:56.3758680Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:56:56.3759065Z 2022-08-17T12:56:56.3939862Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:56:56.3940841Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:56:56.3941463Z Traceback (most recent call last): 2022-08-17T12:56:56.3942394Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:56:56.3943174Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:56:56.3957083Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:56:56.3957557Z 2022-08-17T12:56:56.3958118Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:56:56.3958932Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:56:56.3959501Z Traceback (most recent call last): 2022-08-17T12:56:56.3960416Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:56:56.3961189Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:56:56.3961796Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:56:56.3962176Z 2022-08-17T12:56:56.4040184Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:56:56.4041025Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:56:56.4041626Z Traceback (most recent call last): 2022-08-17T12:56:56.4042543Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:56:56.4043293Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:56:56.4043950Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:56:56.4044318Z 2022-08-17T12:56:56.8075286Z ok (2.479s) 2022-08-17T12:56:56.8075537Z 2022-08-17T12:56:56.8076017Z ---------------------------------------------------------------------- 2022-08-17T12:56:56.8076436Z Ran 1 test in 2.479s 2022-08-17T12:56:56.8076622Z 2022-08-17T12:56:56.8076721Z OK 2022-08-17T12:56:56.8076873Z 2022-08-17T12:56:56.8077014Z Generating XML reports... 2022-08-17T12:56:56.8110577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125654.xml 2022-08-17T12:56:58.1071324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:58.1071798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:58.1074233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:58.1074616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:58.2589032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcpg3juz 2022-08-17T12:56:58.2592354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcpg3juz/_remote_module_non_scriptable.py 2022-08-17T12:56:58.5281961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:56:58.5292961Z 2022-08-17T12:56:58.5293428Z Running tests... 2022-08-17T12:56:58.5294108Z ---------------------------------------------------------------------- 2022-08-17T12:56:58.7906707Z test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27858 2022-08-17T12:56:58.7936979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27859 2022-08-17T12:56:58.7965768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27860 2022-08-17T12:56:58.7996592Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27861 2022-08-17T12:56:59.7850535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:59.7851207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:59.7854264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:59.7854925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:59.8437700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:59.8438267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:59.8442104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:59.8442758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:59.8515040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:59.8515744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:59.8519699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:59.8520340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:59.8598057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:56:59.8598723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:56:59.8601621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:56:59.8602304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:56:59.9437865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo4zv4ao 2022-08-17T12:56:59.9438676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo4zv4ao/_remote_module_non_scriptable.py 2022-08-17T12:57:00.0057075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bw_9cz2 2022-08-17T12:57:00.0058346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bw_9cz2/_remote_module_non_scriptable.py 2022-08-17T12:57:00.0114104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckwkvdcp 2022-08-17T12:57:00.0115456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckwkvdcp/_remote_module_non_scriptable.py 2022-08-17T12:57:00.0192620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4zxkfm7 2022-08-17T12:57:00.0194765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4zxkfm7/_remote_module_non_scriptable.py 2022-08-17T12:57:00.2052638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:00.2675556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:00.2742867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:00.2841937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:00.9055090Z ok (2.376s) 2022-08-17T12:57:00.9055294Z 2022-08-17T12:57:00.9056161Z ---------------------------------------------------------------------- 2022-08-17T12:57:00.9056437Z Ran 1 test in 2.376s 2022-08-17T12:57:00.9056559Z 2022-08-17T12:57:00.9056623Z OK 2022-08-17T12:57:00.9056719Z 2022-08-17T12:57:00.9056804Z Generating XML reports... 2022-08-17T12:57:00.9090674Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125658.xml 2022-08-17T12:57:02.2261794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:02.2262193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:02.2265150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:02.2265655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:02.3777056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92wxjagi 2022-08-17T12:57:02.3780354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92wxjagi/_remote_module_non_scriptable.py 2022-08-17T12:57:02.6445494Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:02.6456391Z 2022-08-17T12:57:02.6456559Z Running tests... 2022-08-17T12:57:02.6457497Z ---------------------------------------------------------------------- 2022-08-17T12:57:02.9084672Z test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28081 2022-08-17T12:57:02.9113547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28082 2022-08-17T12:57:02.9141402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28083 2022-08-17T12:57:02.9173681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28084 2022-08-17T12:57:03.9364135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:03.9364841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:03.9367381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:03.9367982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:03.9385878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:03.9386537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:03.9388923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:03.9389570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:03.9390401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:03.9391079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:03.9394387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:03.9395049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:03.9942228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:03.9942837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:03.9946111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:03.9946786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:04.0983302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6j92teo 2022-08-17T12:57:04.0984131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6j92teo/_remote_module_non_scriptable.py 2022-08-17T12:57:04.0987361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jpxnzda 2022-08-17T12:57:04.0989732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jpxnzda/_remote_module_non_scriptable.py 2022-08-17T12:57:04.1007606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb19eomfx 2022-08-17T12:57:04.1009848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb19eomfx/_remote_module_non_scriptable.py 2022-08-17T12:57:04.1520332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6jcdutn 2022-08-17T12:57:04.1521380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6jcdutn/_remote_module_non_scriptable.py 2022-08-17T12:57:04.3619363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:04.3637425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:04.3646636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:04.4133563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:05.0230899Z ok (2.377s) 2022-08-17T12:57:05.0231428Z 2022-08-17T12:57:05.0231934Z ---------------------------------------------------------------------- 2022-08-17T12:57:05.0232333Z Ran 1 test in 2.377s 2022-08-17T12:57:05.0232528Z 2022-08-17T12:57:05.0232629Z OK 2022-08-17T12:57:05.0232773Z 2022-08-17T12:57:05.0232920Z Generating XML reports... 2022-08-17T12:57:05.0269043Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125702.xml 2022-08-17T12:57:06.3143038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:06.3143711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:06.3146700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:06.3147302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:06.4660594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc09uhnx 2022-08-17T12:57:06.4662731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc09uhnx/_remote_module_non_scriptable.py 2022-08-17T12:57:06.7335373Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:06.7346616Z 2022-08-17T12:57:06.7347079Z Running tests... 2022-08-17T12:57:06.7347525Z ---------------------------------------------------------------------- 2022-08-17T12:57:07.0001050Z test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28304 2022-08-17T12:57:07.0030793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28305 2022-08-17T12:57:07.0059369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28306 2022-08-17T12:57:07.0093878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28307 2022-08-17T12:57:08.0967646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:08.0968318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:08.0970030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:08.0970666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:08.1134701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:08.1135285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:08.1138320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:08.1139014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:08.1595134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:08.1595760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:08.1599761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:08.1600399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:08.1646620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:08.1647177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:08.1651855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:08.1652537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:08.2570708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohdnrt13 2022-08-17T12:57:08.2571775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohdnrt13/_remote_module_non_scriptable.py 2022-08-17T12:57:08.2725539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm53e3bzf 2022-08-17T12:57:08.2727213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm53e3bzf/_remote_module_non_scriptable.py 2022-08-17T12:57:08.3164065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleoybrgb 2022-08-17T12:57:08.3165216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleoybrgb/_remote_module_non_scriptable.py 2022-08-17T12:57:08.3222961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxvs6fx7 2022-08-17T12:57:08.3224654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxvs6fx7/_remote_module_non_scriptable.py 2022-08-17T12:57:08.5188241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:08.5330879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:08.5752150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:08.5825020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:08.7752628Z do nothing 2022-08-17T12:57:08.7753262Z do nothing 2022-08-17T12:57:08.7919442Z do nothing 2022-08-17T12:57:08.7921425Z do nothing 2022-08-17T12:57:08.7951081Z do nothing 2022-08-17T12:57:08.7954504Z do nothing 2022-08-17T12:57:08.8018455Z do nothing 2022-08-17T12:57:08.8023855Z do nothing 2022-08-17T12:57:09.2152914Z ok (2.480s) 2022-08-17T12:57:09.2153375Z 2022-08-17T12:57:09.2154361Z ---------------------------------------------------------------------- 2022-08-17T12:57:09.2154715Z Ran 1 test in 2.481s 2022-08-17T12:57:09.2154852Z 2022-08-17T12:57:09.2154921Z OK 2022-08-17T12:57:09.2155020Z 2022-08-17T12:57:09.2155104Z Generating XML reports... 2022-08-17T12:57:09.2190414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125706.xml 2022-08-17T12:57:10.5161465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:10.5162146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:10.5164898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:10.6673586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:10.6674005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51a0waxu 2022-08-17T12:57:10.6676562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51a0waxu/_remote_module_non_scriptable.py 2022-08-17T12:57:10.9340380Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:10.9351022Z 2022-08-17T12:57:10.9351123Z Running tests... 2022-08-17T12:57:10.9351645Z ---------------------------------------------------------------------- 2022-08-17T12:57:11.2006376Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28527 2022-08-17T12:57:11.2037059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28528 2022-08-17T12:57:11.2067756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28529 2022-08-17T12:57:11.2098100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28530 2022-08-17T12:57:12.3096528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:12.3097242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:12.3099967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:12.3100615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:12.3409107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:12.3409690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:12.3412582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:12.3413189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:12.3480356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:12.3481055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:12.3484080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:12.3484713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:12.3568510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:12.3569134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:12.3571823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:12.3572391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:12.4705757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wkk2y59 2022-08-17T12:57:12.4707222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wkk2y59/_remote_module_non_scriptable.py 2022-08-17T12:57:12.5017194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnqe9cof 2022-08-17T12:57:12.5018065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnqe9cof/_remote_module_non_scriptable.py 2022-08-17T12:57:12.5106537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nwieaa9 2022-08-17T12:57:12.5109155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nwieaa9/_remote_module_non_scriptable.py 2022-08-17T12:57:12.5190735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g675tay 2022-08-17T12:57:12.5191679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g675tay/_remote_module_non_scriptable.py 2022-08-17T12:57:12.7359333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:12.7650902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:12.7782167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:12.7846508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:13.0681508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:57:13.0883976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:57:13.0884931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:57:13.0888515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:57:13.0890302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:13.0891605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:13.0986707Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:13.0988222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:13.5161750Z ok (2.581s) 2022-08-17T12:57:13.5162015Z 2022-08-17T12:57:13.5162528Z ---------------------------------------------------------------------- 2022-08-17T12:57:13.5162784Z Ran 1 test in 2.581s 2022-08-17T12:57:13.5162907Z 2022-08-17T12:57:13.5162973Z OK 2022-08-17T12:57:13.5163073Z 2022-08-17T12:57:13.5163169Z Generating XML reports... 2022-08-17T12:57:13.5200242Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125710.xml 2022-08-17T12:57:14.8305164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:14.8305570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:14.8309166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:14.8309573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:14.9829225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l1bvgzc 2022-08-17T12:57:14.9832137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l1bvgzc/_remote_module_non_scriptable.py 2022-08-17T12:57:15.2594431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:15.2604348Z 2022-08-17T12:57:15.2604433Z Running tests... 2022-08-17T12:57:15.2604898Z ---------------------------------------------------------------------- 2022-08-17T12:57:15.5230983Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28762 2022-08-17T12:57:15.5259151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28763 2022-08-17T12:57:15.5287732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28764 2022-08-17T12:57:15.5320188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28765 2022-08-17T12:57:16.5632711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:16.5633378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:16.5634146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:16.5634795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:16.5636649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:16.5637538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:16.5638345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:16.5638932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:16.5639666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:16.5640282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:16.5643632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:16.5644300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:16.5878897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:16.5879571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:16.5882893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:16.5883904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:16.7231612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgwpx3c4 2022-08-17T12:57:16.7232299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tt2uwcj 2022-08-17T12:57:16.7232922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3hmfk8m 2022-08-17T12:57:16.7233648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgwpx3c4/_remote_module_non_scriptable.py 2022-08-17T12:57:16.7234368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tt2uwcj/_remote_module_non_scriptable.py 2022-08-17T12:57:16.7235110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3hmfk8m/_remote_module_non_scriptable.py 2022-08-17T12:57:16.7437515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwen47vn 2022-08-17T12:57:16.7438295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwen47vn/_remote_module_non_scriptable.py 2022-08-17T12:57:16.9825813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:16.9826436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:16.9830892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:17.0024827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:17.2195689Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:57:17.2196665Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-08-17T12:57:17.2197304Z Traceback (most recent call last): 2022-08-17T12:57:17.2198278Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:57:17.2199054Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:57:17.2200121Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-08-17T12:57:17.2200862Z raise ValueError(expected_err_escape) 2022-08-17T12:57:17.2201321Z ValueError: 2022-08-17T12:57:17.2201692Z First line of error 2022-08-17T12:57:17.2202106Z next line of error 2022-08-17T12:57:17.2202501Z last line of error 2022-08-17T12:57:17.2202743Z 2022-08-17T12:57:17.2381270Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:57:17.2382314Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-08-17T12:57:17.2382950Z Traceback (most recent call last): 2022-08-17T12:57:17.2384253Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:57:17.2385096Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:57:17.2386198Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-08-17T12:57:17.2386946Z raise ValueError(expected_err_escape) 2022-08-17T12:57:17.2387388Z ValueError: 2022-08-17T12:57:17.2387788Z First line of error 2022-08-17T12:57:17.2388203Z next line of error 2022-08-17T12:57:17.2388588Z last line of error 2022-08-17T12:57:17.2388838Z 2022-08-17T12:57:17.2395725Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:57:17.2396580Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-08-17T12:57:17.2397189Z Traceback (most recent call last): 2022-08-17T12:57:17.2398077Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:57:17.2398860Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:57:17.2399928Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-08-17T12:57:17.2400661Z raise ValueError(expected_err_escape) 2022-08-17T12:57:17.2401332Z ValueError: 2022-08-17T12:57:17.2401730Z First line of error 2022-08-17T12:57:17.2402147Z next line of error 2022-08-17T12:57:17.2402525Z last line of error 2022-08-17T12:57:17.2402771Z 2022-08-17T12:57:17.2496937Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:57:17.2497737Z ValueError('\nFirst line of error \n next line of error \n last line of error') 2022-08-17T12:57:17.2498161Z Traceback (most recent call last): 2022-08-17T12:57:17.2498785Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:57:17.2499304Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:57:17.2500060Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func_escape 2022-08-17T12:57:17.2500631Z raise ValueError(expected_err_escape) 2022-08-17T12:57:17.2500986Z ValueError: 2022-08-17T12:57:17.2501278Z First line of error 2022-08-17T12:57:17.2501548Z next line of error 2022-08-17T12:57:17.2501828Z last line of error 2022-08-17T12:57:17.2502007Z 2022-08-17T12:57:17.6379890Z ok (2.377s) 2022-08-17T12:57:17.6380179Z 2022-08-17T12:57:17.6380662Z ---------------------------------------------------------------------- 2022-08-17T12:57:17.6381076Z Ran 1 test in 2.377s 2022-08-17T12:57:17.6381259Z 2022-08-17T12:57:17.6381357Z OK 2022-08-17T12:57:17.6381495Z 2022-08-17T12:57:17.6381652Z Generating XML reports... 2022-08-17T12:57:17.6415747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125715.xml 2022-08-17T12:57:18.9248165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:18.9248890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:18.9251828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:18.9252447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:19.0740634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd40blgs4 2022-08-17T12:57:19.0744501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd40blgs4/_remote_module_non_scriptable.py 2022-08-17T12:57:19.3538103Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:19.3549060Z 2022-08-17T12:57:19.3549173Z Running tests... 2022-08-17T12:57:19.3549873Z ---------------------------------------------------------------------- 2022-08-17T12:57:19.6268113Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28985 2022-08-17T12:57:19.6299119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28986 2022-08-17T12:57:19.6330129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28987 2022-08-17T12:57:19.6363302Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28988 2022-08-17T12:57:20.6526974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:20.6527635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:20.6530769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:20.6531412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:20.6633879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:20.6634533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:20.6637623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:20.6638523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:20.7062388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:20.7063018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:20.7066317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:20.7066957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:20.7192645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:20.7193365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:20.7196749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:20.7197359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:20.8136613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt494xz2g 2022-08-17T12:57:20.8137622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt494xz2g/_remote_module_non_scriptable.py 2022-08-17T12:57:20.8235300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mhauvaa 2022-08-17T12:57:20.8236503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mhauvaa/_remote_module_non_scriptable.py 2022-08-17T12:57:20.8674715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgv9wfsh 2022-08-17T12:57:20.8675519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgv9wfsh/_remote_module_non_scriptable.py 2022-08-17T12:57:20.8792938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17bz6dyj 2022-08-17T12:57:20.8793826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17bz6dyj/_remote_module_non_scriptable.py 2022-08-17T12:57:21.0795506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:21.0856833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:21.1310112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:21.1402912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:21.8424592Z ok (2.487s) 2022-08-17T12:57:21.8424867Z 2022-08-17T12:57:21.8425220Z ---------------------------------------------------------------------- 2022-08-17T12:57:21.8425762Z Ran 1 test in 2.487s 2022-08-17T12:57:21.8425884Z 2022-08-17T12:57:21.8425959Z OK 2022-08-17T12:57:21.8426057Z 2022-08-17T12:57:21.8426152Z Generating XML reports... 2022-08-17T12:57:21.8462304Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125719.xml 2022-08-17T12:57:23.1365061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:23.1365462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:23.1367833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:23.1368369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:23.2862750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqcl_zw_ 2022-08-17T12:57:23.2866254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqcl_zw_/_remote_module_non_scriptable.py 2022-08-17T12:57:23.5504218Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:23.5514686Z 2022-08-17T12:57:23.5514837Z Running tests... 2022-08-17T12:57:23.5515765Z ---------------------------------------------------------------------- 2022-08-17T12:57:23.8122230Z test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29208 2022-08-17T12:57:23.8152380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29209 2022-08-17T12:57:23.8181621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29210 2022-08-17T12:57:23.8214115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29211 2022-08-17T12:57:24.8467906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:24.8468574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:24.8471773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:24.8472475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:24.8745490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:24.8746125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:24.8749505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:24.8750213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:24.9023995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:24.9024612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:24.9027315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:24.9027885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:24.9100792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:24.9101318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:24.9105110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:24.9105800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:25.0073053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4onhc3g 2022-08-17T12:57:25.0073887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4onhc3g/_remote_module_non_scriptable.py 2022-08-17T12:57:25.0321689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6e6sfl8 2022-08-17T12:57:25.0322510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6e6sfl8/_remote_module_non_scriptable.py 2022-08-17T12:57:25.0598477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4bofdde 2022-08-17T12:57:25.0599382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4bofdde/_remote_module_non_scriptable.py 2022-08-17T12:57:25.0676228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnscp3nc 2022-08-17T12:57:25.0677580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnscp3nc/_remote_module_non_scriptable.py 2022-08-17T12:57:25.2668675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:25.2930756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:25.3183667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:25.3273514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:26.0273326Z ok (2.475s) 2022-08-17T12:57:26.0273546Z 2022-08-17T12:57:26.0274127Z ---------------------------------------------------------------------- 2022-08-17T12:57:26.0274403Z Ran 1 test in 2.476s 2022-08-17T12:57:26.0274529Z 2022-08-17T12:57:26.0274595Z OK 2022-08-17T12:57:26.0274695Z 2022-08-17T12:57:26.0274792Z Generating XML reports... 2022-08-17T12:57:26.0307444Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125723.xml 2022-08-17T12:57:27.3340443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:27.3340820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:27.3343675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:27.3344077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:27.4861874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cgvodqp 2022-08-17T12:57:27.4865464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cgvodqp/_remote_module_non_scriptable.py 2022-08-17T12:57:27.7576182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:27.7586365Z 2022-08-17T12:57:27.7586504Z Running tests... 2022-08-17T12:57:27.7587122Z ---------------------------------------------------------------------- 2022-08-17T12:57:28.0235896Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29431 2022-08-17T12:57:28.0264175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29432 2022-08-17T12:57:28.0293192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29433 2022-08-17T12:57:28.0324146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29434 2022-08-17T12:57:29.1354349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:29.1355027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:29.1357761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:29.1358412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:29.1516538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:29.1517205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:29.1520196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:29.1521164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:29.1592983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:29.1593648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:29.1596987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:29.1597611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:29.1680868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:29.1681499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:29.1684756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:29.1685427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:29.2960831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz1z6g5j 2022-08-17T12:57:29.2962457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz1z6g5j/_remote_module_non_scriptable.py 2022-08-17T12:57:29.3100269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdid5vv0 2022-08-17T12:57:29.3101530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdid5vv0/_remote_module_non_scriptable.py 2022-08-17T12:57:29.3182813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qczec6c 2022-08-17T12:57:29.3184798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qczec6c/_remote_module_non_scriptable.py 2022-08-17T12:57:29.3259240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunla5ex2 2022-08-17T12:57:29.3260330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunla5ex2/_remote_module_non_scriptable.py 2022-08-17T12:57:29.5579416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:29.5696186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:29.5777493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:29.5868223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:30.2385680Z ok (2.479s) 2022-08-17T12:57:30.2385964Z 2022-08-17T12:57:30.2386438Z ---------------------------------------------------------------------- 2022-08-17T12:57:30.2386875Z Ran 1 test in 2.480s 2022-08-17T12:57:30.2387064Z 2022-08-17T12:57:30.2387168Z OK 2022-08-17T12:57:30.2387303Z 2022-08-17T12:57:30.2387437Z Generating XML reports... 2022-08-17T12:57:30.2420442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125727.xml 2022-08-17T12:57:31.5392025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:31.5392414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:31.5394424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:31.5394999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:31.6906213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwv1tmqfr 2022-08-17T12:57:31.6909325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwv1tmqfr/_remote_module_non_scriptable.py 2022-08-17T12:57:31.9548247Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:31.9558188Z 2022-08-17T12:57:31.9558325Z Running tests... 2022-08-17T12:57:31.9559188Z ---------------------------------------------------------------------- 2022-08-17T12:57:32.2175019Z test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29654 2022-08-17T12:57:32.2203266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29655 2022-08-17T12:57:32.2232224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29656 2022-08-17T12:57:32.2263684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29657 2022-08-17T12:57:33.2410909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:33.2411638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:33.2414847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:33.2415546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:33.2573741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:33.2574419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:33.2577578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:33.2578271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:33.2671388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:33.2672079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:33.2675480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:33.2676137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:33.2699612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:33.2700252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:33.2705027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:33.2705700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:33.4019230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6kz_vmp 2022-08-17T12:57:33.4020037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6kz_vmp/_remote_module_non_scriptable.py 2022-08-17T12:57:33.4159790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24f5pc44 2022-08-17T12:57:33.4161874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24f5pc44/_remote_module_non_scriptable.py 2022-08-17T12:57:33.4272698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1xaj204 2022-08-17T12:57:33.4273712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1xaj204/_remote_module_non_scriptable.py 2022-08-17T12:57:33.4307695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf4vqnx4 2022-08-17T12:57:33.4309479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf4vqnx4/_remote_module_non_scriptable.py 2022-08-17T12:57:33.6634080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:33.6750285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:33.6889312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:33.6937679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:34.3322281Z ok (2.376s) 2022-08-17T12:57:34.3322520Z 2022-08-17T12:57:34.3323362Z ---------------------------------------------------------------------- 2022-08-17T12:57:34.3323630Z Ran 1 test in 2.376s 2022-08-17T12:57:34.3323757Z 2022-08-17T12:57:34.3323821Z OK 2022-08-17T12:57:34.3323903Z 2022-08-17T12:57:34.3323999Z Generating XML reports... 2022-08-17T12:57:34.3357790Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125731.xml 2022-08-17T12:57:35.6397096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:35.6397489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:35.6400488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:35.6400879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:35.7902377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2wb0le5 2022-08-17T12:57:35.7906672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2wb0le5/_remote_module_non_scriptable.py 2022-08-17T12:57:36.0609136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:36.0619708Z 2022-08-17T12:57:36.0620128Z Running tests... 2022-08-17T12:57:36.0620780Z ---------------------------------------------------------------------- 2022-08-17T12:57:36.3276173Z test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29877 2022-08-17T12:57:36.3304908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29878 2022-08-17T12:57:36.3334765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29879 2022-08-17T12:57:36.3368665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29880 2022-08-17T12:57:37.4487129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:37.4487837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:37.4489550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:37.4490158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:37.4558567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:37.4559213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:37.4562616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:37.4563268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:37.4634613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:37.4635304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:37.4639486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:37.4640169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:37.4684481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:37.4685089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:37.4688616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:37.4689288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:37.6070360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdahx72mo 2022-08-17T12:57:37.6071909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdahx72mo/_remote_module_non_scriptable.py 2022-08-17T12:57:37.6135864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3rv58xm 2022-08-17T12:57:37.6138008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3rv58xm/_remote_module_non_scriptable.py 2022-08-17T12:57:37.6222439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprd99n3qa 2022-08-17T12:57:37.6223553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprd99n3qa/_remote_module_non_scriptable.py 2022-08-17T12:57:37.6281976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7mui61r 2022-08-17T12:57:37.6283524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7mui61r/_remote_module_non_scriptable.py 2022-08-17T12:57:37.8668771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:37.8711897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:37.8847285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:37.8916611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:38.5429023Z ok (2.481s) 2022-08-17T12:57:38.5429611Z 2022-08-17T12:57:38.5430145Z ---------------------------------------------------------------------- 2022-08-17T12:57:38.5430619Z Ran 1 test in 2.481s 2022-08-17T12:57:38.5430814Z 2022-08-17T12:57:38.5430892Z OK 2022-08-17T12:57:38.5431072Z 2022-08-17T12:57:38.5431242Z Generating XML reports... 2022-08-17T12:57:38.5463809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125736.xml 2022-08-17T12:57:39.8557022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:39.8557710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:39.8560394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:39.8560998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:40.0122679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps30r3t_f 2022-08-17T12:57:40.0125294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps30r3t_f/_remote_module_non_scriptable.py 2022-08-17T12:57:40.2731831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:40.2743069Z 2022-08-17T12:57:40.2743490Z Running tests... 2022-08-17T12:57:40.2743916Z ---------------------------------------------------------------------- 2022-08-17T12:57:40.5345040Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30100 2022-08-17T12:57:40.5373251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30101 2022-08-17T12:57:40.5401606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30102 2022-08-17T12:57:40.5432836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30103 2022-08-17T12:57:41.5456737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:41.5457342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:41.5458121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:41.5458707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:41.5459510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:41.5460074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:41.5461208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:41.5461806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:41.5918509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:41.5919161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:41.5922517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:41.5923166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:41.6078201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:41.6078886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:41.6082683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:41.6083245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:41.7039059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9ymsucb 2022-08-17T12:57:41.7039908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9ymsucb/_remote_module_non_scriptable.py 2022-08-17T12:57:41.7055123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9c52t7o 2022-08-17T12:57:41.7056576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9c52t7o/_remote_module_non_scriptable.py 2022-08-17T12:57:41.7485802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ouaakhx 2022-08-17T12:57:41.7486814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ouaakhx/_remote_module_non_scriptable.py 2022-08-17T12:57:41.7650188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3onyf0a_ 2022-08-17T12:57:41.7651040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3onyf0a_/_remote_module_non_scriptable.py 2022-08-17T12:57:41.9623724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:41.9684276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:42.0075041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:42.0232184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:42.7493558Z ok (2.475s) 2022-08-17T12:57:42.7493847Z 2022-08-17T12:57:42.7494349Z ---------------------------------------------------------------------- 2022-08-17T12:57:42.7494603Z Ran 1 test in 2.475s 2022-08-17T12:57:42.7494727Z 2022-08-17T12:57:42.7494792Z OK 2022-08-17T12:57:42.7494888Z 2022-08-17T12:57:42.7495008Z Generating XML reports... 2022-08-17T12:57:42.7529860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125740.xml 2022-08-17T12:57:44.0395964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:44.0396384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:44.0399075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:44.0399450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:44.1949688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5q6j4n8 2022-08-17T12:57:44.1953387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5q6j4n8/_remote_module_non_scriptable.py 2022-08-17T12:57:44.4586217Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:44.4597029Z 2022-08-17T12:57:44.4597131Z Running tests... 2022-08-17T12:57:44.4597637Z ---------------------------------------------------------------------- 2022-08-17T12:57:44.7215427Z test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30323 2022-08-17T12:57:44.7245738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30324 2022-08-17T12:57:44.7276060Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30325 2022-08-17T12:57:44.7308714Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30326 2022-08-17T12:57:45.7505065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:45.7505721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:45.7508893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:45.7509579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:45.8537218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:45.8538206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:45.8540751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:45.8541421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:45.8542244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:45.8542783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:45.8547327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:45.8548162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:45.8630573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:45.8631264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:45.8635419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:45.8636081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:45.9081648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8qa24vj 2022-08-17T12:57:45.9083173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8qa24vj/_remote_module_non_scriptable.py 2022-08-17T12:57:46.0113972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply1up0cb 2022-08-17T12:57:46.0114824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply1up0cb/_remote_module_non_scriptable.py 2022-08-17T12:57:46.0126528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7_lxu1u 2022-08-17T12:57:46.0129015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7_lxu1u/_remote_module_non_scriptable.py 2022-08-17T12:57:46.0220804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypflxx5z 2022-08-17T12:57:46.0221923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypflxx5z/_remote_module_non_scriptable.py 2022-08-17T12:57:46.1679892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:46.2707208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:46.2723609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:46.2850314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:46.9369294Z ok (2.477s) 2022-08-17T12:57:46.9369541Z 2022-08-17T12:57:46.9369870Z ---------------------------------------------------------------------- 2022-08-17T12:57:46.9370123Z Ran 1 test in 2.477s 2022-08-17T12:57:46.9370248Z 2022-08-17T12:57:46.9370312Z OK 2022-08-17T12:57:46.9370424Z 2022-08-17T12:57:46.9370519Z Generating XML reports... 2022-08-17T12:57:46.9403341Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125744.xml 2022-08-17T12:57:48.2125619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:48.2126534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:48.2128665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:48.2129270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:48.3644381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lvo6ya_ 2022-08-17T12:57:48.3646831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lvo6ya_/_remote_module_non_scriptable.py 2022-08-17T12:57:48.6260801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:48.6272073Z 2022-08-17T12:57:48.6272566Z Running tests... 2022-08-17T12:57:48.6272997Z ---------------------------------------------------------------------- 2022-08-17T12:57:48.8932821Z test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30546 2022-08-17T12:57:48.8961187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30547 2022-08-17T12:57:48.8990476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30548 2022-08-17T12:57:48.9022902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30549 2022-08-17T12:57:49.9929507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:49.9930190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:49.9933686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:49.9934345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:50.0224314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:50.0224970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:50.0228182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:50.0228852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:50.0235109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:50.0235766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:50.0240319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:50.0240981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:50.0408856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:50.0409527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:50.0412600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:50.0413297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:50.1512623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5amfydz 2022-08-17T12:57:50.1514317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5amfydz/_remote_module_non_scriptable.py 2022-08-17T12:57:50.1821578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxupwi7jn 2022-08-17T12:57:50.1823226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxupwi7jn/_remote_module_non_scriptable.py 2022-08-17T12:57:50.1827528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeim2ivbf 2022-08-17T12:57:50.1829532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeim2ivbf/_remote_module_non_scriptable.py 2022-08-17T12:57:50.2001707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi3esq70 2022-08-17T12:57:50.2002739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi3esq70/_remote_module_non_scriptable.py 2022-08-17T12:57:50.4147450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:50.4434691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:50.4435518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:50.4836570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:51.1082399Z ok (2.481s) 2022-08-17T12:57:51.1082634Z 2022-08-17T12:57:51.1083103Z ---------------------------------------------------------------------- 2022-08-17T12:57:51.1083509Z Ran 1 test in 2.481s 2022-08-17T12:57:51.1083700Z 2022-08-17T12:57:51.1083796Z OK 2022-08-17T12:57:51.1083951Z 2022-08-17T12:57:51.1084089Z Generating XML reports... 2022-08-17T12:57:51.1117574Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125748.xml 2022-08-17T12:57:52.4234893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:52.4235375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:52.4236697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:52.4237086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:52.5788208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk4actpr 2022-08-17T12:57:52.5791202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk4actpr/_remote_module_non_scriptable.py 2022-08-17T12:57:52.8430428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:52.8441037Z 2022-08-17T12:57:52.8441185Z Running tests... 2022-08-17T12:57:52.8441838Z ---------------------------------------------------------------------- 2022-08-17T12:57:53.1030995Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30769 2022-08-17T12:57:53.1058202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30770 2022-08-17T12:57:53.1085624Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30771 2022-08-17T12:57:53.1116707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30772 2022-08-17T12:57:54.1348408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:54.1349047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:54.1351547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:54.1352221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:54.1927955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:54.1928988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:54.1930736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:54.1931356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:54.2318591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:54.2319296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:54.2322225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:54.2322914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:54.2609674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:54.2610357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:54.2613425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:54.2614337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:54.2918071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s0zltu4 2022-08-17T12:57:54.2918918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s0zltu4/_remote_module_non_scriptable.py 2022-08-17T12:57:54.3498766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsn5v1db 2022-08-17T12:57:54.3499706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsn5v1db/_remote_module_non_scriptable.py 2022-08-17T12:57:54.3888508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pq3ndxe 2022-08-17T12:57:54.3889423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pq3ndxe/_remote_module_non_scriptable.py 2022-08-17T12:57:54.4181422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu986bscq 2022-08-17T12:57:54.4182874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu986bscq/_remote_module_non_scriptable.py 2022-08-17T12:57:54.5651479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:54.6139784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:54.6488601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:54.6823812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:55.0171171Z ok (2.173s) 2022-08-17T12:57:55.0171384Z 2022-08-17T12:57:55.0171933Z ---------------------------------------------------------------------- 2022-08-17T12:57:55.0172257Z Ran 1 test in 2.173s 2022-08-17T12:57:55.0172379Z 2022-08-17T12:57:55.0172444Z OK 2022-08-17T12:57:55.0172539Z 2022-08-17T12:57:55.0172634Z Generating XML reports... 2022-08-17T12:57:55.0205580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125752.xml 2022-08-17T12:57:56.3261547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:56.3262275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:56.3265239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:56.3265816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:56.4804203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0tbe6rx 2022-08-17T12:57:56.4807832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0tbe6rx/_remote_module_non_scriptable.py 2022-08-17T12:57:56.7524368Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:57:56.7535839Z 2022-08-17T12:57:56.7536203Z Running tests... 2022-08-17T12:57:56.7536748Z ---------------------------------------------------------------------- 2022-08-17T12:57:57.0225795Z test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30828 2022-08-17T12:57:57.0254875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30829 2022-08-17T12:57:57.0283756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30830 2022-08-17T12:57:57.0314915Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30831 2022-08-17T12:57:58.0787524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:58.0788233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:58.0791200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:58.0791866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:58.0858387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:58.0859068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:58.0861929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:58.0862592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:58.1445762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:58.1446637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:58.1448434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:58.1449091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:58.1720294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:57:58.1720970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:57:58.1724005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:57:58.1724660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:57:58.2384353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8txbrot 2022-08-17T12:57:58.2385310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8txbrot/_remote_module_non_scriptable.py 2022-08-17T12:57:58.2432198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeppyzna 2022-08-17T12:57:58.2433293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeppyzna/_remote_module_non_scriptable.py 2022-08-17T12:57:58.3029039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjiliztbh 2022-08-17T12:57:58.3029856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjiliztbh/_remote_module_non_scriptable.py 2022-08-17T12:57:58.3304622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvo0egzz 2022-08-17T12:57:58.3305639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvo0egzz/_remote_module_non_scriptable.py 2022-08-17T12:57:58.5004271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:57:58.5019319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:57:58.5635019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:57:58.5923877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:57:58.9169433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:57:58.9170569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:57:58.9172009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:58.9172949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:57:58.9173808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:57:58.9174969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:58.9176200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:58.9180861Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:57:59.3376680Z ok (2.584s) 2022-08-17T12:57:59.3377238Z 2022-08-17T12:57:59.3377561Z ---------------------------------------------------------------------- 2022-08-17T12:57:59.3377858Z Ran 1 test in 2.584s 2022-08-17T12:57:59.3377979Z 2022-08-17T12:57:59.3378030Z OK 2022-08-17T12:57:59.3378129Z 2022-08-17T12:57:59.3378226Z Generating XML reports... 2022-08-17T12:57:59.3412081Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125756.xml 2022-08-17T12:58:00.6370890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:00.6371289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:00.6372985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:00.6373358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:00.7871756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yfvcizq 2022-08-17T12:58:00.7875073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yfvcizq/_remote_module_non_scriptable.py 2022-08-17T12:58:01.0479739Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:01.0490134Z 2022-08-17T12:58:01.0490273Z Running tests... 2022-08-17T12:58:01.0490871Z ---------------------------------------------------------------------- 2022-08-17T12:58:01.3096411Z test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31063 2022-08-17T12:58:01.3124448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31064 2022-08-17T12:58:01.3152540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31065 2022-08-17T12:58:01.3185249Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31066 2022-08-17T12:58:02.3379965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:02.3380603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:02.3383883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:02.3384520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:02.3463014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:02.3463654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:02.3467096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:02.3467767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:02.3853343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:02.3854044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:02.3855247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:02.3855939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:02.3897304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:02.3897987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:02.3902607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:02.3903329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:02.4967084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2h4bephj 2022-08-17T12:58:02.4968083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2h4bephj/_remote_module_non_scriptable.py 2022-08-17T12:58:02.5028928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm3xi1lk 2022-08-17T12:58:02.5030414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm3xi1lk/_remote_module_non_scriptable.py 2022-08-17T12:58:02.5449997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo7bkwny 2022-08-17T12:58:02.5450790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo7bkwny/_remote_module_non_scriptable.py 2022-08-17T12:58:02.5472865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztwuqi5g 2022-08-17T12:58:02.5474401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztwuqi5g/_remote_module_non_scriptable.py 2022-08-17T12:58:02.7567983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:02.7606568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:02.8062647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:02.8069936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:03.5244677Z ok (2.475s) 2022-08-17T12:58:03.5244964Z 2022-08-17T12:58:03.5245478Z ---------------------------------------------------------------------- 2022-08-17T12:58:03.5245761Z Ran 1 test in 2.475s 2022-08-17T12:58:03.5246036Z 2022-08-17T12:58:03.5246141Z OK 2022-08-17T12:58:03.5246233Z 2022-08-17T12:58:03.5246330Z Generating XML reports... 2022-08-17T12:58:03.5279158Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125801.xml 2022-08-17T12:58:04.8152043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:04.8152442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:04.8155019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:04.8155534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:04.9674131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4jfremn 2022-08-17T12:58:04.9676758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4jfremn/_remote_module_non_scriptable.py 2022-08-17T12:58:05.2284463Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:05.2295085Z 2022-08-17T12:58:05.2295395Z Running tests... 2022-08-17T12:58:05.2296010Z ---------------------------------------------------------------------- 2022-08-17T12:58:05.4908087Z test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31286 2022-08-17T12:58:05.4935286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31287 2022-08-17T12:58:05.4963417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31288 2022-08-17T12:58:05.4995265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31289 2022-08-17T12:58:06.5432024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:06.5432631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:06.5434962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:06.5435557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:06.5524642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:06.5525601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:06.5529247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:06.5529947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:06.6209529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:06.6210210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:06.6213386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:06.6214062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:06.6519396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:06.6520066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:06.6522909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:06.6523606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:06.7018044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjjwot8y 2022-08-17T12:58:06.7019630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjjwot8y/_remote_module_non_scriptable.py 2022-08-17T12:58:06.7102331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cxkw_o6 2022-08-17T12:58:06.7103128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cxkw_o6/_remote_module_non_scriptable.py 2022-08-17T12:58:06.7788997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35_pvpj8 2022-08-17T12:58:06.7789832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35_pvpj8/_remote_module_non_scriptable.py 2022-08-17T12:58:06.8149921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_hr512b 2022-08-17T12:58:06.8150729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_hr512b/_remote_module_non_scriptable.py 2022-08-17T12:58:06.9669148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:06.9727107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:07.0470770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:07.0787094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:07.3082932Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:58:07.3083903Z ValueError('Expected error') 2022-08-17T12:58:07.3084566Z Traceback (most recent call last): 2022-08-17T12:58:07.3086235Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:07.3087178Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:07.3088257Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-08-17T12:58:07.3088956Z raise ValueError("Expected error") 2022-08-17T12:58:07.3089446Z ValueError: Expected error 2022-08-17T12:58:07.3089727Z 2022-08-17T12:58:07.3284834Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:58:07.3285458Z ValueError('Expected error') 2022-08-17T12:58:07.3285780Z Traceback (most recent call last): 2022-08-17T12:58:07.3286727Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:07.3287354Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:07.3288352Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-08-17T12:58:07.3289206Z raise ValueError("Expected error") 2022-08-17T12:58:07.3289603Z ValueError: Expected error 2022-08-17T12:58:07.3289840Z 2022-08-17T12:58:07.3372029Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:58:07.3372718Z ValueError('Expected error') 2022-08-17T12:58:07.3373220Z Traceback (most recent call last): 2022-08-17T12:58:07.3374176Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:07.3374955Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:07.3375994Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-08-17T12:58:07.3376735Z raise ValueError("Expected error") 2022-08-17T12:58:07.3377216Z ValueError: Expected error 2022-08-17T12:58:07.3377498Z 2022-08-17T12:58:07.3408356Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:58:07.3409620Z ValueError('Expected error') 2022-08-17T12:58:07.3410099Z Traceback (most recent call last): 2022-08-17T12:58:07.3410885Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:07.3411453Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:07.3412121Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc 2022-08-17T12:58:07.3412596Z raise ValueError("Expected error") 2022-08-17T12:58:07.3412922Z ValueError: Expected error 2022-08-17T12:58:07.3413112Z 2022-08-17T12:58:07.7057442Z ok (2.476s) 2022-08-17T12:58:07.7057673Z 2022-08-17T12:58:07.7058022Z ---------------------------------------------------------------------- 2022-08-17T12:58:07.7058305Z Ran 1 test in 2.476s 2022-08-17T12:58:07.7058412Z 2022-08-17T12:58:07.7058477Z OK 2022-08-17T12:58:07.7058574Z 2022-08-17T12:58:07.7058672Z Generating XML reports... 2022-08-17T12:58:07.7092038Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125805.xml 2022-08-17T12:58:09.0032886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:09.0033590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:09.0036930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:09.0037530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:09.1571157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88hxaem4 2022-08-17T12:58:09.1574747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88hxaem4/_remote_module_non_scriptable.py 2022-08-17T12:58:09.4187740Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:09.4198969Z 2022-08-17T12:58:09.6800832Z Running tests... 2022-08-17T12:58:09.6801637Z ---------------------------------------------------------------------- 2022-08-17T12:58:09.6802375Z test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31509 2022-08-17T12:58:09.6829719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31510 2022-08-17T12:58:09.6858342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31511 2022-08-17T12:58:09.6890112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31512 2022-08-17T12:58:10.7149811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:10.7150496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:10.7153529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:10.7154433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:10.7253888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:10.7254563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:10.7258027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:10.7258712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:10.7568328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:10.7569045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:10.7571576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:10.7572280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:10.7682110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:10.7682660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:10.7685633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:10.7686407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:10.8749275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwngd3vn9 2022-08-17T12:58:10.8750402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwngd3vn9/_remote_module_non_scriptable.py 2022-08-17T12:58:10.8845001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt19r7lc 2022-08-17T12:58:10.8846277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt19r7lc/_remote_module_non_scriptable.py 2022-08-17T12:58:10.9176248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpj_6e40 2022-08-17T12:58:10.9177056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpj_6e40/_remote_module_non_scriptable.py 2022-08-17T12:58:10.9276659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6ssnizm 2022-08-17T12:58:10.9277537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6ssnizm/_remote_module_non_scriptable.py 2022-08-17T12:58:11.1367296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:11.1444659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:11.1838701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:11.1945182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:11.3753038Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:58:11.3753716Z ValueError('Expected error') 2022-08-17T12:58:11.3754094Z Traceback (most recent call last): 2022-08-17T12:58:11.3754929Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.3755537Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.3756322Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:58:11.3756649Z raise ValueError(expected_err) 2022-08-17T12:58:11.3756865Z ValueError: Expected error 2022-08-17T12:58:11.3757007Z 2022-08-17T12:58:11.3912261Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:58:11.3913369Z ValueError('Expected error') 2022-08-17T12:58:11.3913855Z Traceback (most recent call last): 2022-08-17T12:58:11.3914820Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.3915949Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.3917000Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:58:11.3917702Z raise ValueError(expected_err) 2022-08-17T12:58:11.3918153Z ValueError: Expected error 2022-08-17T12:58:11.3918432Z 2022-08-17T12:58:11.3939989Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:58:11.3941025Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:58:11.3941713Z Traceback (most recent call last): 2022-08-17T12:58:11.3942637Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.3943422Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.3944166Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:58:11.3944544Z 2022-08-17T12:58:11.3973884Z On WorkerInfo(id=0, name=worker0): 2022-08-17T12:58:11.3974426Z ValueError('Expected error') 2022-08-17T12:58:11.3974805Z Traceback (most recent call last): 2022-08-17T12:58:11.3975539Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.3976277Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.3977073Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:58:11.3977627Z raise ValueError(expected_err) 2022-08-17T12:58:11.3977985Z ValueError: Expected error 2022-08-17T12:58:11.3978209Z 2022-08-17T12:58:11.4140614Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:58:11.4141026Z ValueError('Expected error') 2022-08-17T12:58:11.4141895Z Traceback (most recent call last): 2022-08-17T12:58:11.4142942Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.4143723Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.4144835Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T12:58:11.4145517Z raise ValueError(expected_err) 2022-08-17T12:58:11.4145991Z ValueError: Expected error 2022-08-17T12:58:11.4146274Z 2022-08-17T12:58:11.4454206Z On WorkerInfo(id=1, name=worker1): 2022-08-17T12:58:11.4454840Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:58:11.4455209Z Traceback (most recent call last): 2022-08-17T12:58:11.4455653Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.4456281Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.4456571Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:58:11.4456745Z 2022-08-17T12:58:11.4493444Z On WorkerInfo(id=3, name=worker3): 2022-08-17T12:58:11.4494363Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:58:11.4494910Z Traceback (most recent call last): 2022-08-17T12:58:11.4495696Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.4496286Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.4496591Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:58:11.4496748Z 2022-08-17T12:58:11.4641918Z On WorkerInfo(id=2, name=worker2): 2022-08-17T12:58:11.4642827Z TypeError('no_result() takes 0 positional arguments but 1 was given') 2022-08-17T12:58:11.4643427Z Traceback (most recent call last): 2022-08-17T12:58:11.4644352Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T12:58:11.4645133Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T12:58:11.4646144Z TypeError: no_result() takes 0 positional arguments but 1 was given 2022-08-17T12:58:11.4646540Z 2022-08-17T12:58:11.8950404Z ok (2.475s) 2022-08-17T12:58:11.8950575Z 2022-08-17T12:58:11.8950892Z ---------------------------------------------------------------------- 2022-08-17T12:58:11.8951167Z Ran 1 test in 2.475s 2022-08-17T12:58:11.8951316Z 2022-08-17T12:58:11.8951422Z OK 2022-08-17T12:58:11.8951519Z 2022-08-17T12:58:11.8951613Z Generating XML reports... 2022-08-17T12:58:11.8984855Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125809.xml 2022-08-17T12:58:13.2091002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:13.2091412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:13.2094013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:13.2094405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:13.3636090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ovex_62 2022-08-17T12:58:13.3639343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ovex_62/_remote_module_non_scriptable.py 2022-08-17T12:58:13.6281121Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:13.6292060Z 2022-08-17T12:58:13.6292161Z Running tests... 2022-08-17T12:58:13.6292638Z ---------------------------------------------------------------------- 2022-08-17T12:58:13.8981701Z test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31732 2022-08-17T12:58:13.9010800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31733 2022-08-17T12:58:13.9040293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31734 2022-08-17T12:58:13.9071313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31735 2022-08-17T12:58:14.9745432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:14.9746086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:14.9747179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:14.9747828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:14.9888644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:14.9889638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:14.9892722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:14.9893427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:15.0328507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:15.0329174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:15.0332128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:15.0332784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:15.0538719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:15.0539364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:15.0541972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:15.0542567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:15.1324560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3932_36j 2022-08-17T12:58:15.1325382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3932_36j/_remote_module_non_scriptable.py 2022-08-17T12:58:15.1476995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp444zowia 2022-08-17T12:58:15.1478217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp444zowia/_remote_module_non_scriptable.py 2022-08-17T12:58:15.1930417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph98wvfns 2022-08-17T12:58:15.1931514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph98wvfns/_remote_module_non_scriptable.py 2022-08-17T12:58:15.2108400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv__33szz 2022-08-17T12:58:15.2109221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv__33szz/_remote_module_non_scriptable.py 2022-08-17T12:58:15.3932925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:15.4083736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:15.4557199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:15.4930697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:16.1132283Z ok (2.484s) 2022-08-17T12:58:16.1132517Z 2022-08-17T12:58:16.1132897Z ---------------------------------------------------------------------- 2022-08-17T12:58:16.1133168Z Ran 1 test in 2.484s 2022-08-17T12:58:16.1133359Z 2022-08-17T12:58:16.1133452Z OK 2022-08-17T12:58:16.1133548Z 2022-08-17T12:58:16.1133647Z Generating XML reports... 2022-08-17T12:58:16.1165810Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125813.xml 2022-08-17T12:58:17.4156999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:17.4157401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:17.4159337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:17.4159766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:17.5772612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9eocn7e 2022-08-17T12:58:17.5775697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9eocn7e/_remote_module_non_scriptable.py 2022-08-17T12:58:17.8389724Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:17.8400196Z 2022-08-17T12:58:17.8400442Z Running tests... 2022-08-17T12:58:17.8401070Z ---------------------------------------------------------------------- 2022-08-17T12:58:18.0973522Z test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31955 2022-08-17T12:58:18.1000172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31956 2022-08-17T12:58:18.1028788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31957 2022-08-17T12:58:18.1060657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31958 2022-08-17T12:58:19.1139948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:19.1140623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:19.1144144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:19.1241485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:19.1242793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:19.1243374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:19.1246358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:19.1246975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:19.1409989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:19.1410660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:19.1413807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:19.1414469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:19.1477863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:19.1478506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:19.1481999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:19.1482657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:19.2742077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_jy_f7q 2022-08-17T12:58:19.2742895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_jy_f7q/_remote_module_non_scriptable.py 2022-08-17T12:58:19.2825120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hpnphzy 2022-08-17T12:58:19.2826488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hpnphzy/_remote_module_non_scriptable.py 2022-08-17T12:58:19.3002825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj62agxgp 2022-08-17T12:58:19.3003638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj62agxgp/_remote_module_non_scriptable.py 2022-08-17T12:58:19.3066067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10l0ptz8 2022-08-17T12:58:19.3067520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10l0ptz8/_remote_module_non_scriptable.py 2022-08-17T12:58:19.5380133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:19.5450767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:19.5642941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:19.5726536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:20.2118990Z ok (2.372s) 2022-08-17T12:58:20.2119864Z 2022-08-17T12:58:20.2120503Z ---------------------------------------------------------------------- 2022-08-17T12:58:20.2121007Z Ran 1 test in 2.372s 2022-08-17T12:58:20.2121201Z 2022-08-17T12:58:20.2121270Z OK 2022-08-17T12:58:20.2121367Z 2022-08-17T12:58:20.2121449Z Generating XML reports... 2022-08-17T12:58:20.2153899Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125817.xml 2022-08-17T12:58:21.4922261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:21.4922662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:21.4925273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:21.4925777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:21.6425104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfxoqr09 2022-08-17T12:58:21.6428392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfxoqr09/_remote_module_non_scriptable.py 2022-08-17T12:58:21.9033364Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:21.9043879Z 2022-08-17T12:58:21.9044025Z Running tests... 2022-08-17T12:58:21.9044662Z ---------------------------------------------------------------------- 2022-08-17T12:58:22.1621865Z test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32178 2022-08-17T12:58:22.1649024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32179 2022-08-17T12:58:22.1677177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32180 2022-08-17T12:58:22.1710804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32181 2022-08-17T12:58:23.1444105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:23.1444782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:23.1448312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:23.1448952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:23.1693927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:23.1694604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:23.1697792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:23.1698615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:23.1913069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:23.1913653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:23.1916824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:23.1917406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:23.2071935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:23.2072480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:23.2076003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:23.2076983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:23.3033522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyvp2hhc 2022-08-17T12:58:23.3034516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyvp2hhc/_remote_module_non_scriptable.py 2022-08-17T12:58:23.3284440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm21t09po 2022-08-17T12:58:23.3285334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm21t09po/_remote_module_non_scriptable.py 2022-08-17T12:58:23.3485778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlt1bft1 2022-08-17T12:58:23.3488237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlt1bft1/_remote_module_non_scriptable.py 2022-08-17T12:58:23.3680404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wp5d4a3 2022-08-17T12:58:23.3681234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wp5d4a3/_remote_module_non_scriptable.py 2022-08-17T12:58:23.5637841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:23.5915886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:23.6068207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:23.6319367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:24.2768701Z ok (2.372s) 2022-08-17T12:58:24.2768961Z 2022-08-17T12:58:24.2769480Z ---------------------------------------------------------------------- 2022-08-17T12:58:24.2769814Z Ran 1 test in 2.372s 2022-08-17T12:58:24.2769936Z 2022-08-17T12:58:24.2770000Z OK 2022-08-17T12:58:24.2770097Z 2022-08-17T12:58:24.2770195Z Generating XML reports... 2022-08-17T12:58:24.2802946Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125821.xml 2022-08-17T12:58:25.5652168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:25.5652564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:25.5654831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:25.5655333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:25.7167649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt54gkh3m 2022-08-17T12:58:25.7171024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt54gkh3m/_remote_module_non_scriptable.py 2022-08-17T12:58:25.9828691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:25.9839293Z 2022-08-17T12:58:25.9839547Z Running tests... 2022-08-17T12:58:25.9840185Z ---------------------------------------------------------------------- 2022-08-17T12:58:26.2505525Z test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32401 2022-08-17T12:58:26.2535643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32402 2022-08-17T12:58:26.2564853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32403 2022-08-17T12:58:26.2597594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32404 2022-08-17T12:58:27.4029181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:27.4029820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:27.4032976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:27.4033588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:27.4089143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:27.4089785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:27.4093619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:27.4094260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:27.4302682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:27.4303322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:27.4305397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:27.4306006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:27.5345077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:27.5345771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:27.5346852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:27.5347745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:27.5642572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyhyjozz 2022-08-17T12:58:27.5643613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyhyjozz/_remote_module_non_scriptable.py 2022-08-17T12:58:27.5692141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf41betz4 2022-08-17T12:58:27.5693510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf41betz4/_remote_module_non_scriptable.py 2022-08-17T12:58:27.5909447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccrz1rug 2022-08-17T12:58:27.5910235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccrz1rug/_remote_module_non_scriptable.py 2022-08-17T12:58:27.6938575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddx280pu 2022-08-17T12:58:27.6939397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddx280pu/_remote_module_non_scriptable.py 2022-08-17T12:58:27.8268640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:27.8330609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:27.8549662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:27.9560868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:28.6660160Z ok (2.682s) 2022-08-17T12:58:28.6660448Z 2022-08-17T12:58:28.6660963Z ---------------------------------------------------------------------- 2022-08-17T12:58:28.6661231Z Ran 1 test in 2.682s 2022-08-17T12:58:28.6661354Z 2022-08-17T12:58:28.6661419Z OK 2022-08-17T12:58:28.6661514Z 2022-08-17T12:58:28.6661611Z Generating XML reports... 2022-08-17T12:58:28.6693856Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125825.xml 2022-08-17T12:58:29.9589760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:29.9590226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:29.9600137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:29.9600503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:30.1178842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp203l24f5 2022-08-17T12:58:30.1181537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp203l24f5/_remote_module_non_scriptable.py 2022-08-17T12:58:30.3786171Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:30.3796916Z 2022-08-17T12:58:30.3797289Z Running tests... 2022-08-17T12:58:30.3797756Z ---------------------------------------------------------------------- 2022-08-17T12:58:30.6414610Z test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32624 2022-08-17T12:58:30.6441681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32625 2022-08-17T12:58:30.6470070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32626 2022-08-17T12:58:30.6501105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32627 2022-08-17T12:58:31.7343286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:31.7343887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:31.7346691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:31.7347585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:31.7497865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:31.7498506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:31.7501160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:31.7501838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:31.7622836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:31.7623498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:31.7628105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:31.7628775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:31.8062758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:31.8063438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:31.8066235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:31.8066944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:31.8911613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zho3a0_ 2022-08-17T12:58:31.8912750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zho3a0_/_remote_module_non_scriptable.py 2022-08-17T12:58:31.9093500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqze60yu8 2022-08-17T12:58:31.9094844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqze60yu8/_remote_module_non_scriptable.py 2022-08-17T12:58:31.9198782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5ddboqw 2022-08-17T12:58:31.9199584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5ddboqw/_remote_module_non_scriptable.py 2022-08-17T12:58:31.9625199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jpaqm3g 2022-08-17T12:58:31.9625976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jpaqm3g/_remote_module_non_scriptable.py 2022-08-17T12:58:32.1498764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:32.1738009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:32.2034496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:32.2212588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:32.8560664Z ok (2.476s) 2022-08-17T12:58:32.8560904Z 2022-08-17T12:58:32.8561411Z ---------------------------------------------------------------------- 2022-08-17T12:58:32.8561802Z Ran 1 test in 2.476s 2022-08-17T12:58:32.8562002Z 2022-08-17T12:58:32.8562105Z OK 2022-08-17T12:58:32.8562249Z 2022-08-17T12:58:32.8562386Z Generating XML reports... 2022-08-17T12:58:32.8596737Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125830.xml 2022-08-17T12:58:34.1702527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:34.1702925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:34.1705427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:34.1705811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:34.3222480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g9nlaf3 2022-08-17T12:58:34.3225417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g9nlaf3/_remote_module_non_scriptable.py 2022-08-17T12:58:34.5833233Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:34.5843418Z 2022-08-17T12:58:34.5843571Z Running tests... 2022-08-17T12:58:34.5844228Z ---------------------------------------------------------------------- 2022-08-17T12:58:34.8460768Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 379 2022-08-17T12:58:34.8489763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 380 2022-08-17T12:58:34.8518782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 381 2022-08-17T12:58:34.8550367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 382 2022-08-17T12:58:35.8583672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:35.8584490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:35.8587843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:35.8588497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:35.8807068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:35.8807741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:35.8810698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:35.8811409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:35.8838265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:35.8838881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:35.8842821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:35.8843458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:35.9237535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:35.9238263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:35.9241553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:35.9242579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:36.0163955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wtxmhdt 2022-08-17T12:58:36.0164874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wtxmhdt/_remote_module_non_scriptable.py 2022-08-17T12:58:36.0415577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitgfs7kd 2022-08-17T12:58:36.0416571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnlybnpd 2022-08-17T12:58:36.0417369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitgfs7kd/_remote_module_non_scriptable.py 2022-08-17T12:58:36.0419639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnlybnpd/_remote_module_non_scriptable.py 2022-08-17T12:58:36.0819819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr87juyqx 2022-08-17T12:58:36.0820658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr87juyqx/_remote_module_non_scriptable.py 2022-08-17T12:58:36.2802814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:36.3046016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:36.3279313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:36.3481768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:36.9607968Z ok (2.376s) 2022-08-17T12:58:36.9608229Z 2022-08-17T12:58:36.9608790Z ---------------------------------------------------------------------- 2022-08-17T12:58:36.9609077Z Ran 1 test in 2.376s 2022-08-17T12:58:36.9609184Z 2022-08-17T12:58:36.9609251Z OK 2022-08-17T12:58:36.9609347Z 2022-08-17T12:58:36.9609442Z Generating XML reports... 2022-08-17T12:58:36.9641535Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125834.xml 2022-08-17T12:58:38.2469215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:38.2469898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:38.2473270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:38.2473895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:38.4001645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvl6hnth9 2022-08-17T12:58:38.4005014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvl6hnth9/_remote_module_non_scriptable.py 2022-08-17T12:58:38.6624644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:38.6635469Z 2022-08-17T12:58:38.6635714Z Running tests... 2022-08-17T12:58:38.6636197Z ---------------------------------------------------------------------- 2022-08-17T12:58:38.9246864Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 614 2022-08-17T12:58:38.9275893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 615 2022-08-17T12:58:38.9305848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 616 2022-08-17T12:58:38.9337069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 617 2022-08-17T12:58:39.9420030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:39.9420626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:39.9423371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:39.9854161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:39.9855199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:39.9855828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:39.9857714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:39.9858364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:39.9900322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:39.9900961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:39.9904546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:39.9905213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:39.9982924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:39.9983616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:39.9988154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:39.9988853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:40.1003583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbirkalt 2022-08-17T12:58:40.1004629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbirkalt/_remote_module_non_scriptable.py 2022-08-17T12:58:40.1444968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex8eeyns 2022-08-17T12:58:40.1446698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex8eeyns/_remote_module_non_scriptable.py 2022-08-17T12:58:40.1470972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9d4yst3 2022-08-17T12:58:40.1472000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9d4yst3/_remote_module_non_scriptable.py 2022-08-17T12:58:40.1556197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw4c_tqe 2022-08-17T12:58:40.1558093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw4c_tqe/_remote_module_non_scriptable.py 2022-08-17T12:58:40.3598928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:40.4038682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:40.4049842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:40.4167753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:41.0395334Z ok (2.376s) 2022-08-17T12:58:41.0395518Z 2022-08-17T12:58:41.0395875Z ---------------------------------------------------------------------- 2022-08-17T12:58:41.0396152Z Ran 1 test in 2.376s 2022-08-17T12:58:41.0396260Z 2022-08-17T12:58:41.0396323Z OK 2022-08-17T12:58:41.0396418Z 2022-08-17T12:58:41.0396528Z Generating XML reports... 2022-08-17T12:58:41.0432269Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125838.xml 2022-08-17T12:58:42.3395225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:42.3395942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:42.3398765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:42.3399346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:42.4948917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_t2xlel 2022-08-17T12:58:42.4953003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_t2xlel/_remote_module_non_scriptable.py 2022-08-17T12:58:42.7639009Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:42.7650459Z 2022-08-17T12:58:42.7650951Z Running tests... 2022-08-17T12:58:42.7651436Z ---------------------------------------------------------------------- 2022-08-17T12:58:43.0315495Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 838 2022-08-17T12:58:43.0343146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 839 2022-08-17T12:58:43.0371282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 840 2022-08-17T12:58:43.0402469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 841 2022-08-17T12:58:44.1210086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:44.1210738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:44.1213833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:44.1214504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:44.1224712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:44.1225308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:44.1226332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:44.1226931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:44.1229663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:44.1230323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:44.1232326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:44.1232976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:44.1334564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:44.1335240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:44.1338801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:44.1339445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:44.2803474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh5wp1zr 2022-08-17T12:58:44.2804310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh5wp1zr/_remote_module_non_scriptable.py 2022-08-17T12:58:44.2841870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hfk_lcm 2022-08-17T12:58:44.2842686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdnmwd3v 2022-08-17T12:58:44.2843661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hfk_lcm/_remote_module_non_scriptable.py 2022-08-17T12:58:44.2845219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdnmwd3v/_remote_module_non_scriptable.py 2022-08-17T12:58:44.2931752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyyo1rbs 2022-08-17T12:58:44.2932807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyyo1rbs/_remote_module_non_scriptable.py 2022-08-17T12:58:44.5397101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:44.5451857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:44.5458669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:44.5539243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:45.2462034Z ok (2.481s) 2022-08-17T12:58:45.2462342Z 2022-08-17T12:58:45.2462885Z ---------------------------------------------------------------------- 2022-08-17T12:58:45.2463336Z Ran 1 test in 2.481s 2022-08-17T12:58:45.2463531Z 2022-08-17T12:58:45.2478855Z OK 2022-08-17T12:58:45.2479208Z 2022-08-17T12:58:45.2479395Z Generating XML reports... 2022-08-17T12:58:45.2504460Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125842.xml 2022-08-17T12:58:46.5400783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:46.5401198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:46.5403815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:46.5404550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:46.6928250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiltu_mw8 2022-08-17T12:58:46.6931731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiltu_mw8/_remote_module_non_scriptable.py 2022-08-17T12:58:46.9600122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:46.9610424Z 2022-08-17T12:58:46.9610542Z Running tests... 2022-08-17T12:58:46.9610970Z ---------------------------------------------------------------------- 2022-08-17T12:58:47.2227540Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1061 2022-08-17T12:58:47.2255581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1062 2022-08-17T12:58:47.2283480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1063 2022-08-17T12:58:47.2315820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1064 2022-08-17T12:58:48.1976715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:48.1977359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:48.1980844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:48.1981533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:48.1987835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:48.1988436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:48.1992777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:48.1993383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:48.2066373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:48.2067030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:48.2070256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:48.2070907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:48.2138867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:48.2139497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:48.2143232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:48.2143899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:48.3598965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc5j0t2h 2022-08-17T12:58:48.3599659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydqq4h2p 2022-08-17T12:58:48.3600844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc5j0t2h/_remote_module_non_scriptable.py 2022-08-17T12:58:48.3601592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydqq4h2p/_remote_module_non_scriptable.py 2022-08-17T12:58:48.3665280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnsqblwq 2022-08-17T12:58:48.3666857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnsqblwq/_remote_module_non_scriptable.py 2022-08-17T12:58:48.3729142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8puk5wj 2022-08-17T12:58:48.3730971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8puk5wj/_remote_module_non_scriptable.py 2022-08-17T12:58:48.6219725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:48.6241894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:48.6298228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:48.6361028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:48.8960713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:58:48.9060534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:58:48.9061795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:58:48.9070666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:58:48.9072069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:48.9169591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:48.9170896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:48.9171750Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:50.3390036Z ok (3.378s) 2022-08-17T12:58:50.3390252Z 2022-08-17T12:58:50.3390693Z ---------------------------------------------------------------------- 2022-08-17T12:58:50.3391038Z Ran 1 test in 3.378s 2022-08-17T12:58:50.3391161Z 2022-08-17T12:58:50.3391229Z OK 2022-08-17T12:58:50.3391325Z 2022-08-17T12:58:50.3394915Z Generating XML reports... 2022-08-17T12:58:50.3427537Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125846.xml 2022-08-17T12:58:51.6640028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:51.6640421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:51.6643475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:51.6644069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:51.8144677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsnvv50p 2022-08-17T12:58:51.8147855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsnvv50p/_remote_module_non_scriptable.py 2022-08-17T12:58:52.0784559Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:52.0794264Z 2022-08-17T12:58:52.0794655Z Running tests... 2022-08-17T12:58:52.0795093Z ---------------------------------------------------------------------- 2022-08-17T12:58:52.3412985Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1296 2022-08-17T12:58:52.3442898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1297 2022-08-17T12:58:52.3472073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1298 2022-08-17T12:58:52.3504136Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1299 2022-08-17T12:58:53.3567845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:53.3568532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:53.3570815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:53.3571752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:53.3654159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:53.3654817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:53.3658332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:53.3658992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:53.3796629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:53.3797296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:53.3800381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:53.3801000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:53.3904892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:53.3905545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:53.3908996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:53.3909651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:53.5155005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1dracxv8 2022-08-17T12:58:53.5155829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1dracxv8/_remote_module_non_scriptable.py 2022-08-17T12:58:53.5231046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsy494te 2022-08-17T12:58:53.5232328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsy494te/_remote_module_non_scriptable.py 2022-08-17T12:58:53.5391041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_c6ig2z 2022-08-17T12:58:53.5392267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_c6ig2z/_remote_module_non_scriptable.py 2022-08-17T12:58:53.5483479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply5g3sbb 2022-08-17T12:58:53.5485594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply5g3sbb/_remote_module_non_scriptable.py 2022-08-17T12:58:53.7741816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:53.7774846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:53.8024223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:53.8073455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:54.0557524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:58:54.0659528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:58:54.0759726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:58:54.0761764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:54.0765815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:58:54.0767833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:54.0769119Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:54.0770353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:58:55.4578543Z ok (3.378s) 2022-08-17T12:58:55.4578755Z 2022-08-17T12:58:55.4579078Z ---------------------------------------------------------------------- 2022-08-17T12:58:55.4579412Z Ran 1 test in 3.378s 2022-08-17T12:58:55.4579538Z 2022-08-17T12:58:55.4579589Z OK 2022-08-17T12:58:55.4579685Z 2022-08-17T12:58:55.4579786Z Generating XML reports... 2022-08-17T12:58:55.4613391Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125852.xml 2022-08-17T12:58:56.7565165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:56.7565561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:56.7567887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:56.7568266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:56.9069418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp190eisu7 2022-08-17T12:58:56.9072893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp190eisu7/_remote_module_non_scriptable.py 2022-08-17T12:58:57.1695154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:58:57.1706257Z 2022-08-17T12:58:57.1706429Z Running tests... 2022-08-17T12:58:57.1706892Z ---------------------------------------------------------------------- 2022-08-17T12:58:57.4364946Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1532 2022-08-17T12:58:57.4392568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1533 2022-08-17T12:58:57.4420524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1534 2022-08-17T12:58:57.4452690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1535 2022-08-17T12:58:58.5031990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:58.5032650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:58.5035749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:58.5036356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:58.5127707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:58.5128665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:58.5132176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:58.5132849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:58.6013403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:58.6014089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:58.6017661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:58.6018505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:58.6051839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:58:58.6052517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:58:58.6056166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:58:58.6056818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:58:58.6606695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgajfaexm 2022-08-17T12:58:58.6607724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgajfaexm/_remote_module_non_scriptable.py 2022-08-17T12:58:58.6698604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wrg4o1l 2022-08-17T12:58:58.6699754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wrg4o1l/_remote_module_non_scriptable.py 2022-08-17T12:58:58.7591067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0c26924 2022-08-17T12:58:58.7591817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0c26924/_remote_module_non_scriptable.py 2022-08-17T12:58:58.7615244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkl50v7ha 2022-08-17T12:58:58.7616276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkl50v7ha/_remote_module_non_scriptable.py 2022-08-17T12:58:58.9215072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:58:58.9284686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:58:59.0187526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:58:59.0211382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:58:59.6512207Z ok (2.480s) 2022-08-17T12:58:59.6512425Z 2022-08-17T12:58:59.6512755Z ---------------------------------------------------------------------- 2022-08-17T12:58:59.6513098Z Ran 1 test in 2.481s 2022-08-17T12:58:59.6513222Z 2022-08-17T12:58:59.6513302Z OK 2022-08-17T12:58:59.6513400Z 2022-08-17T12:58:59.6513483Z Generating XML reports... 2022-08-17T12:58:59.6546520Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125857.xml 2022-08-17T12:59:00.9522262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:00.9522935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:00.9525740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:00.9526574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:01.1053573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf95_9py 2022-08-17T12:59:01.1057280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf95_9py/_remote_module_non_scriptable.py 2022-08-17T12:59:01.3722954Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:01.3735154Z 2022-08-17T12:59:01.3735561Z Running tests... 2022-08-17T12:59:01.3736204Z ---------------------------------------------------------------------- 2022-08-17T12:59:01.6445135Z test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1755 2022-08-17T12:59:01.6473466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1756 2022-08-17T12:59:01.6503951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1757 2022-08-17T12:59:01.6534080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1758 2022-08-17T12:59:02.6705333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:02.6705965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:02.6708825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:02.6709678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:02.6788893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:02.6789618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:02.6792765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:02.6793417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:02.6843856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:02.6844455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:02.6849283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:02.6849940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:02.7344983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:02.7345595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:02.7348889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:02.7349433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:02.8319766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8n1ldlr 2022-08-17T12:59:02.8320495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8n1ldlr/_remote_module_non_scriptable.py 2022-08-17T12:59:02.8387948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacffsg60 2022-08-17T12:59:02.8389387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacffsg60/_remote_module_non_scriptable.py 2022-08-17T12:59:02.8451491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lcc74pg 2022-08-17T12:59:02.8453027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lcc74pg/_remote_module_non_scriptable.py 2022-08-17T12:59:02.8952475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkogdi4_ 2022-08-17T12:59:02.8953521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkogdi4_/_remote_module_non_scriptable.py 2022-08-17T12:59:03.0965974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:03.1004919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:03.1114951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:03.1624671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:03.7593149Z ok (2.385s) 2022-08-17T12:59:03.7593416Z 2022-08-17T12:59:03.7593942Z ---------------------------------------------------------------------- 2022-08-17T12:59:03.7594249Z Ran 1 test in 2.386s 2022-08-17T12:59:03.7594371Z 2022-08-17T12:59:03.7594449Z OK 2022-08-17T12:59:03.7594547Z 2022-08-17T12:59:03.7594649Z Generating XML reports... 2022-08-17T12:59:03.7629672Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125901.xml 2022-08-17T12:59:05.0635714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:05.0636196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:05.0637858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:05.0638427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:05.2176541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoajrsbo 2022-08-17T12:59:05.2179694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoajrsbo/_remote_module_non_scriptable.py 2022-08-17T12:59:05.4874025Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:05.4884332Z 2022-08-17T12:59:05.4884440Z Running tests... 2022-08-17T12:59:05.4884956Z ---------------------------------------------------------------------- 2022-08-17T12:59:05.7599107Z test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1978 2022-08-17T12:59:05.7629098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1979 2022-08-17T12:59:05.7658556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1980 2022-08-17T12:59:05.7690333Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1981 2022-08-17T12:59:06.8129096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:06.8129735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:06.8131996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:06.8132642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:06.8316016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:06.8316632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:06.8320068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:06.8320722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:06.8389265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:06.8389896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:06.8394346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:06.8395019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:06.8402182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:06.8402820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:06.8406315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:06.8406935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:06.9720969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxsgow8_ 2022-08-17T12:59:06.9721983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxsgow8_/_remote_module_non_scriptable.py 2022-08-17T12:59:06.9917692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bmuzwgb 2022-08-17T12:59:06.9918787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bmuzwgb/_remote_module_non_scriptable.py 2022-08-17T12:59:06.9985326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdefooxba 2022-08-17T12:59:06.9986823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdefooxba/_remote_module_non_scriptable.py 2022-08-17T12:59:06.9995649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivs9h8rx 2022-08-17T12:59:06.9997721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivs9h8rx/_remote_module_non_scriptable.py 2022-08-17T12:59:07.2312986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:07.2545810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:07.2590880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:07.2615190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:14.0851193Z ok (8.596s) 2022-08-17T12:59:14.0851523Z 2022-08-17T12:59:14.0852042Z ---------------------------------------------------------------------- 2022-08-17T12:59:14.0852312Z Ran 1 test in 8.597s 2022-08-17T12:59:14.0852435Z 2022-08-17T12:59:14.0852499Z OK 2022-08-17T12:59:14.0852582Z 2022-08-17T12:59:14.0852679Z Generating XML reports... 2022-08-17T12:59:14.0888711Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125905.xml 2022-08-17T12:59:15.3760679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:15.3761387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:15.3763998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:15.3764615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:15.5276379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4c6cufp7 2022-08-17T12:59:15.5280252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4c6cufp7/_remote_module_non_scriptable.py 2022-08-17T12:59:15.7956101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:15.7967853Z 2022-08-17T12:59:15.7968323Z Running tests... 2022-08-17T12:59:15.7968809Z ---------------------------------------------------------------------- 2022-08-17T12:59:16.0651237Z test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2201 2022-08-17T12:59:16.0679924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2202 2022-08-17T12:59:16.0709396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2203 2022-08-17T12:59:16.0741473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2204 2022-08-17T12:59:17.1273220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:17.1273919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:17.1277414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:17.1278091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:17.1391712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:17.1392700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:17.1394080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:17.1394683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:17.1774845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:17.1775493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:17.1778750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:17.1779416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:17.2303637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:17.2304274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:17.2307166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:17.2886246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:17.2887019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ceujh6o 2022-08-17T12:59:17.2887784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ceujh6o/_remote_module_non_scriptable.py 2022-08-17T12:59:17.2990606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36fhjqag 2022-08-17T12:59:17.2992063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36fhjqag/_remote_module_non_scriptable.py 2022-08-17T12:59:17.3417517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehc94eau 2022-08-17T12:59:17.3418407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehc94eau/_remote_module_non_scriptable.py 2022-08-17T12:59:17.3893414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkoba4io 2022-08-17T12:59:17.3894534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkoba4io/_remote_module_non_scriptable.py 2022-08-17T12:59:17.5563225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:17.5676406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:17.6266350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:17.6579635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:17.8961741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:59:17.8963586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:59:17.8964482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:59:17.8966074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:17.8967667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:17.8968575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:59:17.8969747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:17.9063492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:18.5807746Z ok (2.784s) 2022-08-17T12:59:18.5807913Z 2022-08-17T12:59:18.5808551Z ---------------------------------------------------------------------- 2022-08-17T12:59:18.5808837Z Ran 1 test in 2.784s 2022-08-17T12:59:18.5808962Z 2022-08-17T12:59:18.5809025Z OK 2022-08-17T12:59:18.5809151Z 2022-08-17T12:59:18.5809295Z Generating XML reports... 2022-08-17T12:59:18.5841769Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125915.xml 2022-08-17T12:59:19.9080241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:19.9080651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:19.9083459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:19.9083849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:20.0591733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr05m8vhk 2022-08-17T12:59:20.0594720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr05m8vhk/_remote_module_non_scriptable.py 2022-08-17T12:59:20.3213413Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:20.3223576Z 2022-08-17T12:59:20.3223955Z Running tests... 2022-08-17T12:59:20.3224358Z ---------------------------------------------------------------------- 2022-08-17T12:59:20.5880777Z test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2436 2022-08-17T12:59:20.5909315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2437 2022-08-17T12:59:20.5938875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2438 2022-08-17T12:59:20.5972013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2439 2022-08-17T12:59:21.6301427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:21.6302072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:21.6304932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:21.6305596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:21.6711797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:21.6712446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:21.6715140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:21.6715778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:21.6957364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:21.6958069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:21.6960820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:21.6961560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:21.7133502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:21.7134172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:21.7136928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:21.7137609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:21.7886200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7365zli 2022-08-17T12:59:21.7887373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7365zli/_remote_module_non_scriptable.py 2022-08-17T12:59:21.8305206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp332abj7_ 2022-08-17T12:59:21.8306241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp332abj7_/_remote_module_non_scriptable.py 2022-08-17T12:59:21.8538782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_u1cix8 2022-08-17T12:59:21.8540178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_u1cix8/_remote_module_non_scriptable.py 2022-08-17T12:59:21.8714670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv51enb50 2022-08-17T12:59:21.8715692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv51enb50/_remote_module_non_scriptable.py 2022-08-17T12:59:22.0524589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:22.0920920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:22.1145538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:22.1311255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:22.8032757Z ok (2.481s) 2022-08-17T12:59:22.8033583Z 2022-08-17T12:59:22.8034234Z ---------------------------------------------------------------------- 2022-08-17T12:59:22.8034712Z Ran 1 test in 2.481s 2022-08-17T12:59:22.8034930Z 2022-08-17T12:59:22.8035032Z OK 2022-08-17T12:59:22.8035132Z 2022-08-17T12:59:22.8035230Z Generating XML reports... 2022-08-17T12:59:22.8067869Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125920.xml 2022-08-17T12:59:24.1103538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:24.1104223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:24.1107017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:24.1107608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:24.2616458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49v5ryex 2022-08-17T12:59:24.2620168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49v5ryex/_remote_module_non_scriptable.py 2022-08-17T12:59:24.5240660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:24.5251762Z 2022-08-17T12:59:24.5252075Z Running tests... 2022-08-17T12:59:24.5252744Z ---------------------------------------------------------------------- 2022-08-17T12:59:24.7884880Z test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2659 2022-08-17T12:59:24.7913902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2660 2022-08-17T12:59:24.7944164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2661 2022-08-17T12:59:24.7977053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2662 2022-08-17T12:59:25.7912015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:25.7912648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:25.7915048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:25.7915706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:25.8192346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:25.8193001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:25.8194402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:25.8195042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:25.8313522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:25.8317218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:25.8318057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:25.8318710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:25.8589006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:25.8589672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:25.8591703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:25.8592375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:25.9522598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0h0adyp 2022-08-17T12:59:25.9523874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0h0adyp/_remote_module_non_scriptable.py 2022-08-17T12:59:25.9783925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y70pj6x 2022-08-17T12:59:25.9785178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y70pj6x/_remote_module_non_scriptable.py 2022-08-17T12:59:25.9920249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94ymwriy 2022-08-17T12:59:25.9921294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94ymwriy/_remote_module_non_scriptable.py 2022-08-17T12:59:26.0167508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaof7k5j1 2022-08-17T12:59:26.0168314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaof7k5j1/_remote_module_non_scriptable.py 2022-08-17T12:59:26.2140074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:26.2384009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:26.2514590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:26.2778525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:26.9033861Z ok (2.378s) 2022-08-17T12:59:26.9034037Z 2022-08-17T12:59:26.9034531Z ---------------------------------------------------------------------- 2022-08-17T12:59:26.9035018Z Ran 1 test in 2.378s 2022-08-17T12:59:26.9035179Z 2022-08-17T12:59:26.9035938Z OK 2022-08-17T12:59:26.9036108Z 2022-08-17T12:59:26.9036213Z Generating XML reports... 2022-08-17T12:59:26.9071329Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125924.xml 2022-08-17T12:59:28.1977210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:28.1977632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:28.1979849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:28.1980242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:28.3508690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnnkz_r5 2022-08-17T12:59:28.3511753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnnkz_r5/_remote_module_non_scriptable.py 2022-08-17T12:59:28.6109927Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:28.6120615Z 2022-08-17T12:59:28.6120742Z Running tests... 2022-08-17T12:59:28.6121363Z ---------------------------------------------------------------------- 2022-08-17T12:59:28.8728355Z test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2882 2022-08-17T12:59:28.8756538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2883 2022-08-17T12:59:28.8789392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2884 2022-08-17T12:59:28.8820022Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2885 2022-08-17T12:59:29.9533489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:29.9534130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:29.9535829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:29.9536501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:29.9739935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:29.9740862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:29.9743068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:29.9743734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:29.9962804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:29.9963441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:29.9966793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:30.0158331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:30.0159297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:30.0159887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:30.0162201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:30.0162830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:30.1118924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ilf1wbz 2022-08-17T12:59:30.1119763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ilf1wbz/_remote_module_non_scriptable.py 2022-08-17T12:59:30.1329177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bdnqpmq 2022-08-17T12:59:30.1330193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bdnqpmq/_remote_module_non_scriptable.py 2022-08-17T12:59:30.1549356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvb8h2gn 2022-08-17T12:59:30.1550065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvb8h2gn/_remote_module_non_scriptable.py 2022-08-17T12:59:30.1746143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpioouicin 2022-08-17T12:59:30.1746977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpioouicin/_remote_module_non_scriptable.py 2022-08-17T12:59:30.3715295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:30.3964073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:30.4167608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:30.4378505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:30.6955896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T12:59:30.7056623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T12:59:30.7057811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T12:59:30.7059344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:30.7061212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T12:59:30.7062397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:30.7063639Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:30.7158235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T12:59:30.7524353Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:59:30.7526360Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1) 2022-08-17T12:59:30.7526866Z 2022-08-17T12:59:30.7540853Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:59:30.7543145Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1) 2022-08-17T12:59:30.7543758Z 2022-08-17T12:59:30.7553245Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:59:30.7554979Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1) 2022-08-17T12:59:30.7555487Z 2022-08-17T12:59:30.7585233Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 2022-08-17T12:59:30.7586242Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1) 2022-08-17T12:59:30.7586526Z 2022-08-17T12:59:31.1880804Z ok (2.576s) 2022-08-17T12:59:31.1880953Z 2022-08-17T12:59:31.1881282Z ---------------------------------------------------------------------- 2022-08-17T12:59:31.1881874Z Ran 1 test in 2.576s 2022-08-17T12:59:31.1881998Z 2022-08-17T12:59:31.1882064Z OK 2022-08-17T12:59:31.1882163Z 2022-08-17T12:59:31.1882249Z Generating XML reports... 2022-08-17T12:59:31.1914849Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125928.xml 2022-08-17T12:59:32.4955482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:32.4955859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:32.4958233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:32.4958612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:32.6442713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_id03rq 2022-08-17T12:59:32.6446386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_id03rq/_remote_module_non_scriptable.py 2022-08-17T12:59:32.9076807Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:32.9087846Z 2022-08-17T12:59:32.9087995Z Running tests... 2022-08-17T12:59:32.9088871Z ---------------------------------------------------------------------- 2022-08-17T12:59:33.1733176Z test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3117 2022-08-17T12:59:33.1762058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3118 2022-08-17T12:59:33.1791494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3119 2022-08-17T12:59:33.1821951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3120 2022-08-17T12:59:34.1732310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:34.1733486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:34.1734520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:34.1735139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:34.1735947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:34.1736600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:34.1740252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:34.1740900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:34.2236864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:34.2237403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:34.2240396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:34.2240939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:34.2317869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:34.2318411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:34.2322436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:34.2322993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:34.3319948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9co8bjty 2022-08-17T12:59:34.3321111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9co8bjty/_remote_module_non_scriptable.py 2022-08-17T12:59:34.3327647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposs8sg2a 2022-08-17T12:59:34.3330261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposs8sg2a/_remote_module_non_scriptable.py 2022-08-17T12:59:34.3829891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ns9hmf4 2022-08-17T12:59:34.3831132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ns9hmf4/_remote_module_non_scriptable.py 2022-08-17T12:59:34.3894662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hi0yc7o 2022-08-17T12:59:34.3896027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hi0yc7o/_remote_module_non_scriptable.py 2022-08-17T12:59:34.5892974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:34.5917965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:34.6450584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:34.6487110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:35.3881643Z ok (2.479s) 2022-08-17T12:59:35.3881904Z 2022-08-17T12:59:35.3882451Z ---------------------------------------------------------------------- 2022-08-17T12:59:35.3883018Z Ran 1 test in 2.479s 2022-08-17T12:59:35.3883141Z 2022-08-17T12:59:35.3883208Z OK 2022-08-17T12:59:35.3883305Z 2022-08-17T12:59:35.3883403Z Generating XML reports... 2022-08-17T12:59:35.3917259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125932.xml 2022-08-17T12:59:36.7121625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:36.7122018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:36.7124653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:36.7125080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:36.8631576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59_7nmjh 2022-08-17T12:59:36.8634853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59_7nmjh/_remote_module_non_scriptable.py 2022-08-17T12:59:37.1263517Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:37.1274525Z 2022-08-17T12:59:37.1274622Z Running tests... 2022-08-17T12:59:37.1275198Z ---------------------------------------------------------------------- 2022-08-17T12:59:37.3911606Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3340 2022-08-17T12:59:37.3939328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3341 2022-08-17T12:59:37.3967577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3342 2022-08-17T12:59:37.3998485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3343 2022-08-17T12:59:38.4040053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:38.4040704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:38.4042052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:38.4042645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:38.4080446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:38.4081246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:38.4084665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:38.4085674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:38.4360084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:38.4360715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:38.4363945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:38.4364597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:38.4700773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:38.4701448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:38.4704677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:38.4705232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:38.5642010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k4lpik6 2022-08-17T12:59:38.5642883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k4lpik6/_remote_module_non_scriptable.py 2022-08-17T12:59:38.5681097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5y1nmfn 2022-08-17T12:59:38.5682321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5y1nmfn/_remote_module_non_scriptable.py 2022-08-17T12:59:38.5957135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52ahji64 2022-08-17T12:59:38.5957995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52ahji64/_remote_module_non_scriptable.py 2022-08-17T12:59:38.6303089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31ha_jjg 2022-08-17T12:59:38.6304655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31ha_jjg/_remote_module_non_scriptable.py 2022-08-17T12:59:38.8258439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:38.8288734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:38.8590623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:38.8974471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:39.5057089Z ok (2.378s) 2022-08-17T12:59:39.5057355Z 2022-08-17T12:59:39.5057889Z ---------------------------------------------------------------------- 2022-08-17T12:59:39.5058170Z Ran 1 test in 2.378s 2022-08-17T12:59:39.5058381Z 2022-08-17T12:59:39.5058436Z OK 2022-08-17T12:59:39.5058539Z 2022-08-17T12:59:39.5058635Z Generating XML reports... 2022-08-17T12:59:39.5091900Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125937.xml 2022-08-17T12:59:40.7988678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:40.7989088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:40.7991318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:40.7991800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:40.9506026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0zjxgq5 2022-08-17T12:59:40.9509207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0zjxgq5/_remote_module_non_scriptable.py 2022-08-17T12:59:41.2136020Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:41.2146552Z 2022-08-17T12:59:41.2146686Z Running tests... 2022-08-17T12:59:41.2147603Z ---------------------------------------------------------------------- 2022-08-17T12:59:41.4808870Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3563 2022-08-17T12:59:41.4838179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3564 2022-08-17T12:59:41.4866747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3565 2022-08-17T12:59:41.4897541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3566 2022-08-17T12:59:42.5376027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:42.5376650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:42.5379143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:42.5379758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:42.5414956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:42.5415559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:42.5419948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:42.5420549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:42.5542044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:42.5545148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:42.5546040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:42.5546662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:42.5725439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:42.5726420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:42.5729053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:42.5729727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:42.6971699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0onvsjl_ 2022-08-17T12:59:42.6972539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0onvsjl_/_remote_module_non_scriptable.py 2022-08-17T12:59:42.7014254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf2d0lbz 2022-08-17T12:59:42.7015738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf2d0lbz/_remote_module_non_scriptable.py 2022-08-17T12:59:42.7129148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjpt1di0 2022-08-17T12:59:42.7130439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjpt1di0/_remote_module_non_scriptable.py 2022-08-17T12:59:42.7329847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6adxlk8 2022-08-17T12:59:42.7331103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6adxlk8/_remote_module_non_scriptable.py 2022-08-17T12:59:42.9592069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:42.9603573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:42.9699317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:42.9958525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:43.6959955Z ok (2.481s) 2022-08-17T12:59:43.6960237Z 2022-08-17T12:59:43.6961059Z ---------------------------------------------------------------------- 2022-08-17T12:59:43.6961322Z Ran 1 test in 2.481s 2022-08-17T12:59:43.6961446Z 2022-08-17T12:59:43.6961511Z OK 2022-08-17T12:59:43.6961606Z 2022-08-17T12:59:43.6961741Z Generating XML reports... 2022-08-17T12:59:43.6994292Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125941.xml 2022-08-17T12:59:44.9466081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:44.9466805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:44.9470242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:44.9470884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:45.0965426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6q355zd 2022-08-17T12:59:45.0969892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6q355zd/_remote_module_non_scriptable.py 2022-08-17T12:59:45.3573002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:45.3584290Z 2022-08-17T12:59:45.3584810Z Running tests... 2022-08-17T12:59:45.3585463Z ---------------------------------------------------------------------- 2022-08-17T12:59:45.6185503Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3786 2022-08-17T12:59:45.6213855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3787 2022-08-17T12:59:45.6246651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3788 2022-08-17T12:59:45.6278404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3789 2022-08-17T12:59:46.6348747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:46.6349357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:46.6352313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:46.6352975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:46.6749543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:46.6750204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:46.6752610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:46.7142414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:46.7143447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:46.7144078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:46.7145028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:46.7145616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:46.7146463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:46.7147123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:46.7149330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:46.7149963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:46.7937056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwxu9li3 2022-08-17T12:59:46.7937893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwxu9li3/_remote_module_non_scriptable.py 2022-08-17T12:59:46.8453139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylayihe2 2022-08-17T12:59:46.8454150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylayihe2/_remote_module_non_scriptable.py 2022-08-17T12:59:46.8726341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxqrrplc 2022-08-17T12:59:46.8727738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxqrrplc/_remote_module_non_scriptable.py 2022-08-17T12:59:46.8742970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9cczoz5 2022-08-17T12:59:46.8745771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9cczoz5/_remote_module_non_scriptable.py 2022-08-17T12:59:47.0520796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:47.1064906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:47.1348661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:47.1356035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:47.8337030Z ok (2.475s) 2022-08-17T12:59:47.8337203Z 2022-08-17T12:59:47.8337934Z ---------------------------------------------------------------------- 2022-08-17T12:59:47.8338195Z Ran 1 test in 2.475s 2022-08-17T12:59:47.8338316Z 2022-08-17T12:59:47.8338382Z OK 2022-08-17T12:59:47.8338480Z 2022-08-17T12:59:47.8338576Z Generating XML reports... 2022-08-17T12:59:47.8371333Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125945.xml 2022-08-17T12:59:49.1170891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:49.1171298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:49.1173605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:49.1173985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:49.2667688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplodugwb3 2022-08-17T12:59:49.2671016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplodugwb3/_remote_module_non_scriptable.py 2022-08-17T12:59:49.5295871Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:49.5306864Z 2022-08-17T12:59:49.5307186Z Running tests... 2022-08-17T12:59:49.5307596Z ---------------------------------------------------------------------- 2022-08-17T12:59:49.7925217Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4009 2022-08-17T12:59:49.7956585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4010 2022-08-17T12:59:49.7989777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4011 2022-08-17T12:59:49.8022806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4012 2022-08-17T12:59:50.8059290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:50.8059950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:50.8061900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:50.8062519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:50.8213165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:50.8213743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:50.8216730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:50.8217422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:50.8256731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:50.8257506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:50.8287448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:50.8288160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:50.8479402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:50.8480092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:50.8483923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:50.8484663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:50.9668182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p2p_qrn 2022-08-17T12:59:50.9669266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p2p_qrn/_remote_module_non_scriptable.py 2022-08-17T12:59:50.9834569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42vojym9 2022-08-17T12:59:50.9835525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42vojym9/_remote_module_non_scriptable.py 2022-08-17T12:59:50.9866027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrn_y8i_ 2022-08-17T12:59:50.9868265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrn_y8i_/_remote_module_non_scriptable.py 2022-08-17T12:59:51.0203345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthemf4e2 2022-08-17T12:59:51.0204345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthemf4e2/_remote_module_non_scriptable.py 2022-08-17T12:59:51.2258820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:51.2449733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:51.2490629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:51.2827128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:51.9080606Z ok (2.377s) 2022-08-17T12:59:51.9080812Z 2022-08-17T12:59:51.9081236Z ---------------------------------------------------------------------- 2022-08-17T12:59:51.9081580Z Ran 1 test in 2.377s 2022-08-17T12:59:51.9081740Z 2022-08-17T12:59:51.9081830Z OK 2022-08-17T12:59:51.9081958Z 2022-08-17T12:59:51.9082065Z Generating XML reports... 2022-08-17T12:59:51.9121590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125949.xml 2022-08-17T12:59:53.2046465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:53.2046872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:53.2049630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:53.2050022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:53.3612054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jwtev0w 2022-08-17T12:59:53.3615538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jwtev0w/_remote_module_non_scriptable.py 2022-08-17T12:59:53.6324496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:53.6335356Z 2022-08-17T12:59:53.6335590Z Running tests... 2022-08-17T12:59:53.6335974Z ---------------------------------------------------------------------- 2022-08-17T12:59:53.9032698Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4232 2022-08-17T12:59:53.9063552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4233 2022-08-17T12:59:53.9092731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4234 2022-08-17T12:59:53.9125195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4235 2022-08-17T12:59:54.9602296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:54.9602955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:54.9605472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:54.9606299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:54.9926796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:54.9927732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:54.9930277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:54.9930895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:55.0420314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:55.0420953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:55.0424640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:55.0425292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:55.0703797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:55.0704548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:55.0708012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:55.0708639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:55.1396373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64pdr05c 2022-08-17T12:59:55.1397540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64pdr05c/_remote_module_non_scriptable.py 2022-08-17T12:59:55.1515175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ij7gw4j 2022-08-17T12:59:55.1516002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ij7gw4j/_remote_module_non_scriptable.py 2022-08-17T12:59:55.2014969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpr6c2hx 2022-08-17T12:59:55.2016847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpr6c2hx/_remote_module_non_scriptable.py 2022-08-17T12:59:55.2295820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7_c3gxr 2022-08-17T12:59:55.2296660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7_c3gxr/_remote_module_non_scriptable.py 2022-08-17T12:59:55.4006782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:55.4097049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:55.4604420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:55.4909819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T12:59:56.2185164Z ok (2.585s) 2022-08-17T12:59:56.2185401Z 2022-08-17T12:59:56.2185847Z ---------------------------------------------------------------------- 2022-08-17T12:59:56.2186269Z Ran 1 test in 2.585s 2022-08-17T12:59:56.2186459Z 2022-08-17T12:59:56.2186533Z OK 2022-08-17T12:59:56.2186692Z 2022-08-17T12:59:56.2186861Z Generating XML reports... 2022-08-17T12:59:56.2225113Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125953.xml 2022-08-17T12:59:57.5123091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:57.5123588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:57.5126267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:57.5126669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:57.6616317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8illvc5v 2022-08-17T12:59:57.6619751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8illvc5v/_remote_module_non_scriptable.py 2022-08-17T12:59:57.9230811Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T12:59:57.9241077Z 2022-08-17T12:59:57.9241160Z Running tests... 2022-08-17T12:59:57.9241687Z ---------------------------------------------------------------------- 2022-08-17T12:59:58.1913723Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4455 2022-08-17T12:59:58.1943057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4456 2022-08-17T12:59:58.1972109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4457 2022-08-17T12:59:58.2002261Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4458 2022-08-17T12:59:59.1818985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:59.1819693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:59.1823129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:59.1823794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:59.2099181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:59.2099897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:59.2103056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:59.2103645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:59.2177689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:59.2178312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:59.2181926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:59.2182584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:59.2423326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T12:59:59.2423992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T12:59:59.2427083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T12:59:59.2427816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T12:59:59.3407550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgsodm1c 2022-08-17T12:59:59.3408674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgsodm1c/_remote_module_non_scriptable.py 2022-08-17T12:59:59.3680334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuooodg7k 2022-08-17T12:59:59.3681785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuooodg7k/_remote_module_non_scriptable.py 2022-08-17T12:59:59.3790567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrrmqf3g 2022-08-17T12:59:59.3791602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrrmqf3g/_remote_module_non_scriptable.py 2022-08-17T12:59:59.3990183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsep532wr 2022-08-17T12:59:59.3991373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsep532wr/_remote_module_non_scriptable.py 2022-08-17T12:59:59.6005075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T12:59:59.6279386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T12:59:59.6429849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T12:59:59.6593357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:00.3061819Z ok (2.382s) 2022-08-17T13:00:00.3062130Z 2022-08-17T13:00:00.3062631Z ---------------------------------------------------------------------- 2022-08-17T13:00:00.3062884Z Ran 1 test in 2.382s 2022-08-17T13:00:00.3063007Z 2022-08-17T13:00:00.3063075Z OK 2022-08-17T13:00:00.3063170Z 2022-08-17T13:00:00.3063270Z Generating XML reports... 2022-08-17T13:00:00.3096563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125957.xml 2022-08-17T13:00:01.6261723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:01.6262148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:01.6263997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:01.6264517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:01.7809931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b6c2bxk 2022-08-17T13:00:01.7814039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b6c2bxk/_remote_module_non_scriptable.py 2022-08-17T13:00:02.0545521Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:02.0557058Z 2022-08-17T13:00:02.0557454Z Running tests... 2022-08-17T13:00:02.0557872Z ---------------------------------------------------------------------- 2022-08-17T13:00:02.3280635Z test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4678 2022-08-17T13:00:02.3309296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4679 2022-08-17T13:00:02.3341192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4680 2022-08-17T13:00:02.3372685Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4681 2022-08-17T13:00:03.3756903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:03.3757607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:03.3761651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:03.3762319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:03.3994750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:03.3995703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:03.3997342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:03.3998016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:03.4030663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:03.4031316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:03.4034955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:03.4035634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:03.4229419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:03.4229977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:03.4233334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:03.4234023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:03.5367517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolj6jtpw 2022-08-17T13:00:03.5368347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolj6jtpw/_remote_module_non_scriptable.py 2022-08-17T13:00:03.5606160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos1gf6va 2022-08-17T13:00:03.5606943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos1gf6va/_remote_module_non_scriptable.py 2022-08-17T13:00:03.5647258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u1pk3r4 2022-08-17T13:00:03.5649564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u1pk3r4/_remote_module_non_scriptable.py 2022-08-17T13:00:03.5825919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr409r4y 2022-08-17T13:00:03.5826766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr409r4y/_remote_module_non_scriptable.py 2022-08-17T13:00:03.7981001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:03.8205262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:03.8277014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:03.8432639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:04.4430922Z ok (2.387s) 2022-08-17T13:00:04.4431232Z 2022-08-17T13:00:04.4431642Z ---------------------------------------------------------------------- 2022-08-17T13:00:04.4431919Z Ran 1 test in 2.387s 2022-08-17T13:00:04.4432062Z 2022-08-17T13:00:04.4432125Z OK 2022-08-17T13:00:04.4432209Z 2022-08-17T13:00:04.4432304Z Generating XML reports... 2022-08-17T13:00:04.4465775Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130002.xml 2022-08-17T13:00:05.7480047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:05.7480800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:05.7483989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:05.7484591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:05.9024024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx4ksu0r 2022-08-17T13:00:05.9028098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx4ksu0r/_remote_module_non_scriptable.py 2022-08-17T13:00:06.1683600Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:06.1695235Z 2022-08-17T13:00:06.1695731Z Running tests... 2022-08-17T13:00:06.1696189Z ---------------------------------------------------------------------- 2022-08-17T13:00:06.4306284Z test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4901 2022-08-17T13:00:06.4336403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4902 2022-08-17T13:00:06.4366733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4903 2022-08-17T13:00:06.4399576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4904 2022-08-17T13:00:07.5028720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:07.5029418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:07.5087292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:07.5087965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:07.5159624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:07.5160020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:07.5163020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:07.5163660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:07.5615889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:07.5616511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:07.5618657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:07.5619246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:07.5911294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:07.5911976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:07.5914867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:07.5915522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:07.6659768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph00xtxxc 2022-08-17T13:00:07.6661220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph00xtxxc/_remote_module_non_scriptable.py 2022-08-17T13:00:07.6784319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnlezy0i 2022-08-17T13:00:07.6785595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnlezy0i/_remote_module_non_scriptable.py 2022-08-17T13:00:07.7209297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqn8vn18 2022-08-17T13:00:07.7210168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqn8vn18/_remote_module_non_scriptable.py 2022-08-17T13:00:07.7494696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs0b3t4x 2022-08-17T13:00:07.7495827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs0b3t4x/_remote_module_non_scriptable.py 2022-08-17T13:00:07.9331750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:07.9427507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:07.9888147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:08.0170743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:10.6492861Z ok (4.479s) 2022-08-17T13:00:10.6493105Z 2022-08-17T13:00:10.6493649Z ---------------------------------------------------------------------- 2022-08-17T13:00:10.6493995Z Ran 1 test in 4.480s 2022-08-17T13:00:10.6494134Z 2022-08-17T13:00:10.6494187Z OK 2022-08-17T13:00:10.6494282Z 2022-08-17T13:00:10.6494377Z Generating XML reports... 2022-08-17T13:00:10.6528128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130006.xml 2022-08-17T13:00:11.9974933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:11.9975572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:11.9977785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:11.9978416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:12.1522172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfns2kxpp 2022-08-17T13:00:12.1526430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfns2kxpp/_remote_module_non_scriptable.py 2022-08-17T13:00:12.4235136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:12.4246703Z 2022-08-17T13:00:12.4247037Z Running tests... 2022-08-17T13:00:12.4247642Z ---------------------------------------------------------------------- 2022-08-17T13:00:12.6935330Z test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5124 2022-08-17T13:00:12.6965408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5125 2022-08-17T13:00:12.6995083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5126 2022-08-17T13:00:12.7027024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5127 2022-08-17T13:00:13.7353635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:13.7354282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:13.7356748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:13.7357422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:13.7449333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:13.7453163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:13.7454015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:13.7454647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:13.7625702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:13.7626331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:13.7628890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:13.7629553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:13.7907422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:13.7908115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:13.7911645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:13.7912336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:13.8965582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dhvuerl 2022-08-17T13:00:13.8966806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dhvuerl/_remote_module_non_scriptable.py 2022-08-17T13:00:13.9049224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw8xzca8 2022-08-17T13:00:13.9050124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw8xzca8/_remote_module_non_scriptable.py 2022-08-17T13:00:13.9417118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tmptgti 2022-08-17T13:00:13.9418399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tmptgti/_remote_module_non_scriptable.py 2022-08-17T13:00:13.9526807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvp_1760 2022-08-17T13:00:13.9527901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvp_1760/_remote_module_non_scriptable.py 2022-08-17T13:00:14.1585101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:14.1642605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:14.2056966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:14.2155715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:14.9094049Z ok (2.484s) 2022-08-17T13:00:14.9094296Z 2022-08-17T13:00:14.9094809Z ---------------------------------------------------------------------- 2022-08-17T13:00:14.9095262Z Ran 1 test in 2.485s 2022-08-17T13:00:14.9095398Z 2022-08-17T13:00:14.9095461Z OK 2022-08-17T13:00:14.9095557Z 2022-08-17T13:00:14.9095653Z Generating XML reports... 2022-08-17T13:00:14.9138946Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130012.xml 2022-08-17T13:00:16.2197121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:16.2197514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:16.2200592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:16.2201096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:16.3725535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ge8ydh9 2022-08-17T13:00:16.3729432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ge8ydh9/_remote_module_non_scriptable.py 2022-08-17T13:00:16.6392249Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:16.6402782Z 2022-08-17T13:00:16.6402922Z Running tests... 2022-08-17T13:00:16.6403548Z ---------------------------------------------------------------------- 2022-08-17T13:00:16.9054694Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5347 2022-08-17T13:00:16.9082885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5348 2022-08-17T13:00:16.9111180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5349 2022-08-17T13:00:16.9142811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5350 2022-08-17T13:00:18.0223693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:18.0224412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:18.0227241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:18.0227830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:18.0608681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:18.0609347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:18.0611651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:18.0612288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:18.0636628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:18.0637311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:18.0641742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:18.0642357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:18.0643134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:18.0643801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:18.0646302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:18.0647213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:18.1869050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy50jzgr5 2022-08-17T13:00:18.1870832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy50jzgr5/_remote_module_non_scriptable.py 2022-08-17T13:00:18.2220383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1av0w1q0 2022-08-17T13:00:18.2221151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1av0w1q0/_remote_module_non_scriptable.py 2022-08-17T13:00:18.2238328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpab88nbwi 2022-08-17T13:00:18.2240443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpab88nbwi/_remote_module_non_scriptable.py 2022-08-17T13:00:18.2254619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc_l880w 2022-08-17T13:00:18.2256682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc_l880w/_remote_module_non_scriptable.py 2022-08-17T13:00:18.4510709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:18.4852858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:18.4871530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:18.4918445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:19.2206261Z ok (2.580s) 2022-08-17T13:00:19.2206525Z 2022-08-17T13:00:19.2207016Z ---------------------------------------------------------------------- 2022-08-17T13:00:19.2207449Z Ran 1 test in 2.580s 2022-08-17T13:00:19.2207641Z 2022-08-17T13:00:19.2207731Z OK 2022-08-17T13:00:19.2207879Z 2022-08-17T13:00:19.2208021Z Generating XML reports... 2022-08-17T13:00:19.2242406Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130016.xml 2022-08-17T13:00:20.5298400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:20.5298827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:20.5300767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:20.5301140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:20.6808946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4er_ffbg 2022-08-17T13:00:20.6812276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4er_ffbg/_remote_module_non_scriptable.py 2022-08-17T13:00:20.9448579Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:20.9459524Z 2022-08-17T13:00:20.9459621Z Running tests... 2022-08-17T13:00:20.9460229Z ---------------------------------------------------------------------- 2022-08-17T13:00:21.2098469Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5570 2022-08-17T13:00:21.2126068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5571 2022-08-17T13:00:21.2159919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5572 2022-08-17T13:00:21.2190838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5573 2022-08-17T13:00:22.2451777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:22.2452446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:22.2454985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:22.2455950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:22.3780966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:22.3781674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:22.3785945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:22.3786612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:22.3990028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:22.3990718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:22.3993354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:22.3993987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:22.4032242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c1io3th 2022-08-17T13:00:22.4034444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c1io3th/_remote_module_non_scriptable.py 2022-08-17T13:00:22.4081838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:22.4082483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:22.4086844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:22.4087630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:22.5382994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60ypbknc 2022-08-17T13:00:22.5384348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60ypbknc/_remote_module_non_scriptable.py 2022-08-17T13:00:22.5596719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkmdc3e3 2022-08-17T13:00:22.5597560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkmdc3e3/_remote_module_non_scriptable.py 2022-08-17T13:00:22.5667000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2aqagwv 2022-08-17T13:00:22.5667817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2aqagwv/_remote_module_non_scriptable.py 2022-08-17T13:00:22.6678095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:22.8024013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:22.8235005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:22.8266457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:23.4252702Z ok (2.479s) 2022-08-17T13:00:23.4252961Z 2022-08-17T13:00:23.4253469Z ---------------------------------------------------------------------- 2022-08-17T13:00:23.4253897Z Ran 1 test in 2.479s 2022-08-17T13:00:23.4254078Z 2022-08-17T13:00:23.4254156Z OK 2022-08-17T13:00:23.4254311Z 2022-08-17T13:00:23.4254459Z Generating XML reports... 2022-08-17T13:00:23.4287957Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130020.xml 2022-08-17T13:00:24.7428796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:24.7429448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:24.7431695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:24.7432326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:24.8972905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zvisie2 2022-08-17T13:00:24.8976437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zvisie2/_remote_module_non_scriptable.py 2022-08-17T13:00:25.1652996Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:25.1664620Z 2022-08-17T13:00:25.1665076Z Running tests... 2022-08-17T13:00:25.1665509Z ---------------------------------------------------------------------- 2022-08-17T13:00:25.4343624Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5793 2022-08-17T13:00:25.4372461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5794 2022-08-17T13:00:25.4402009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5795 2022-08-17T13:00:25.4433956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5796 2022-08-17T13:00:26.4223720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:26.4224523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:26.4229036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:26.4229682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:26.4319336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:26.4320040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:26.4324177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:26.4324846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:26.5157758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:26.5158440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:26.5161126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:26.5161775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:26.5206639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:26.5207297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:26.5211211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:26.5212209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:26.5830721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54uxu9xa 2022-08-17T13:00:26.5831524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54uxu9xa/_remote_module_non_scriptable.py 2022-08-17T13:00:26.5919716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylngi4w7 2022-08-17T13:00:26.5920463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylngi4w7/_remote_module_non_scriptable.py 2022-08-17T13:00:26.6758238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplczp7dmt 2022-08-17T13:00:26.6759116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplczp7dmt/_remote_module_non_scriptable.py 2022-08-17T13:00:26.6794520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_muwuj2q 2022-08-17T13:00:26.6796489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_muwuj2q/_remote_module_non_scriptable.py 2022-08-17T13:00:26.8492010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:26.8566590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:26.9412223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:26.9450024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:27.6494897Z ok (2.483s) 2022-08-17T13:00:27.6495121Z 2022-08-17T13:00:27.6495446Z ---------------------------------------------------------------------- 2022-08-17T13:00:27.6495788Z Ran 1 test in 2.483s 2022-08-17T13:00:27.6495955Z 2022-08-17T13:00:27.6496006Z OK 2022-08-17T13:00:27.6496103Z 2022-08-17T13:00:27.6496201Z Generating XML reports... 2022-08-17T13:00:27.6530266Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130025.xml 2022-08-17T13:00:28.9766565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:28.9767225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:28.9769933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:28.9770525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:29.1286987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_4mdw1g 2022-08-17T13:00:29.1290529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_4mdw1g/_remote_module_non_scriptable.py 2022-08-17T13:00:29.3914667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:29.3925707Z 2022-08-17T13:00:29.3926363Z Running tests... 2022-08-17T13:00:29.3926791Z ---------------------------------------------------------------------- 2022-08-17T13:00:29.6628406Z test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6016 2022-08-17T13:00:29.6657647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6017 2022-08-17T13:00:29.6687965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6018 2022-08-17T13:00:29.6726964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6019 2022-08-17T13:00:30.7120853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:30.7121479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:30.7127068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:30.7128046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:30.7170473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:30.7171110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:30.7175098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:30.7175763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:30.7497715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:30.7498410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:30.7500874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:30.7501489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:30.7622869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:30.7623453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:30.7627546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:30.7628225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:30.8716705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2xvixlf 2022-08-17T13:00:30.8717487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2xvixlf/_remote_module_non_scriptable.py 2022-08-17T13:00:30.8788443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55rnvxja 2022-08-17T13:00:30.8789396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55rnvxja/_remote_module_non_scriptable.py 2022-08-17T13:00:30.9094405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7tvg57j 2022-08-17T13:00:30.9095249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7tvg57j/_remote_module_non_scriptable.py 2022-08-17T13:00:30.9201285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0p43ycg 2022-08-17T13:00:30.9202378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0p43ycg/_remote_module_non_scriptable.py 2022-08-17T13:00:31.1298915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:31.1421352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:31.1694635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:31.1806648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:32.7801040Z ok (3.387s) 2022-08-17T13:00:32.7801331Z 2022-08-17T13:00:32.7801880Z ---------------------------------------------------------------------- 2022-08-17T13:00:32.7802326Z Ran 1 test in 3.387s 2022-08-17T13:00:32.7802548Z 2022-08-17T13:00:32.7802653Z OK 2022-08-17T13:00:32.7802753Z 2022-08-17T13:00:32.7802850Z Generating XML reports... 2022-08-17T13:00:32.7839038Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130029.xml 2022-08-17T13:00:34.0772670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:34.0773066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:34.0775255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:34.0775757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:34.2269860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppelcopo7 2022-08-17T13:00:34.2272989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppelcopo7/_remote_module_non_scriptable.py 2022-08-17T13:00:34.4870486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:34.4881031Z 2022-08-17T13:00:34.4881184Z Running tests... 2022-08-17T13:00:34.4881699Z ---------------------------------------------------------------------- 2022-08-17T13:00:34.7470532Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6239 2022-08-17T13:00:34.7499819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6240 2022-08-17T13:00:34.7527975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6241 2022-08-17T13:00:34.7558519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6242 2022-08-17T13:00:35.8530554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:35.8531117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:35.8533982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:35.8534594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:35.8584899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:35.8585562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:35.8589233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:35.8589880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:35.8931416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:35.8932061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:35.8935183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:35.8935882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:35.9527850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:35.9528483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:35.9531850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:35.9532498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:36.0174082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_l0kiyw 2022-08-17T13:00:36.0174836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_l0kiyw/_remote_module_non_scriptable.py 2022-08-17T13:00:36.0184842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pzsvmnk 2022-08-17T13:00:36.0186905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pzsvmnk/_remote_module_non_scriptable.py 2022-08-17T13:00:36.0501463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifrjvqpx 2022-08-17T13:00:36.0502626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifrjvqpx/_remote_module_non_scriptable.py 2022-08-17T13:00:36.1104692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuab536ik 2022-08-17T13:00:36.1105530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuab536ik/_remote_module_non_scriptable.py 2022-08-17T13:00:36.2786663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:36.2810488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:36.3090679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:36.3754251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:36.6111934Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:36.6112585Z ValueError('Expected error') 2022-08-17T13:00:36.6112975Z Traceback (most recent call last): 2022-08-17T13:00:36.6113749Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6114411Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6116411Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6117133Z raise ValueError(expected_err) 2022-08-17T13:00:36.6117478Z ValueError: Expected error 2022-08-17T13:00:36.6117789Z 2022-08-17T13:00:36.6123324Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:36.6125663Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:36.6127156Z Traceback (most recent call last): 2022-08-17T13:00:36.6127778Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6128137Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6128574Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:36.6128897Z rref_type = type(rref.local_value()) 2022-08-17T13:00:36.6129314Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:36.6129761Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:36.6130053Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:36.6156831Z ValueError('Expected error') 2022-08-17T13:00:36.6157251Z Traceback (most recent call last): 2022-08-17T13:00:36.6158010Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6158639Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6159258Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6159583Z raise ValueError(expected_err) 2022-08-17T13:00:36.6159809Z ValueError: Expected error 2022-08-17T13:00:36.6159940Z 2022-08-17T13:00:36.6159946Z 2022-08-17T13:00:36.6317252Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:36.6317797Z ValueError('Expected error') 2022-08-17T13:00:36.6318194Z Traceback (most recent call last): 2022-08-17T13:00:36.6318965Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6319799Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6320614Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6321181Z raise ValueError(expected_err) 2022-08-17T13:00:36.6323230Z ValueError: Expected error 2022-08-17T13:00:36.6323571Z 2022-08-17T13:00:36.6323854Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:36.6325799Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:36.6327455Z Traceback (most recent call last): 2022-08-17T13:00:36.6328209Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6328811Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6329590Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:36.6330159Z rref_type = type(rref.local_value()) 2022-08-17T13:00:36.6330921Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:36.6331694Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:36.6332232Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:36.6332714Z ValueError('Expected error') 2022-08-17T13:00:36.6333215Z Traceback (most recent call last): 2022-08-17T13:00:36.6333942Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6334573Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6335078Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6335385Z raise ValueError(expected_err) 2022-08-17T13:00:36.6335602Z ValueError: Expected error 2022-08-17T13:00:36.6335729Z 2022-08-17T13:00:36.6335734Z 2022-08-17T13:00:36.6434042Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:36.6434584Z ValueError('Expected error') 2022-08-17T13:00:36.6434996Z Traceback (most recent call last): 2022-08-17T13:00:36.6435752Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6436340Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6437751Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6440042Z raise ValueError(expected_err) 2022-08-17T13:00:36.6440301Z ValueError: Expected error 2022-08-17T13:00:36.6440431Z 2022-08-17T13:00:36.6440527Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:36.6441607Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:36.6442301Z Traceback (most recent call last): 2022-08-17T13:00:36.6442712Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6443063Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6443493Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:36.6443804Z rref_type = type(rref.local_value()) 2022-08-17T13:00:36.6444219Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:36.6444671Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:36.6444961Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:36.6445406Z ValueError('Expected error') 2022-08-17T13:00:36.6445627Z Traceback (most recent call last): 2022-08-17T13:00:36.6446199Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6446554Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6447023Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6447343Z raise ValueError(expected_err) 2022-08-17T13:00:36.6447546Z ValueError: Expected error 2022-08-17T13:00:36.6447676Z 2022-08-17T13:00:36.6448007Z 2022-08-17T13:00:36.6518577Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:36.6518932Z ValueError('Expected error') 2022-08-17T13:00:36.6519222Z Traceback (most recent call last): 2022-08-17T13:00:36.6519652Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6520079Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6520552Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6520940Z raise ValueError(expected_err) 2022-08-17T13:00:36.6521381Z ValueError: Expected error 2022-08-17T13:00:36.6521576Z 2022-08-17T13:00:36.6545317Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:36.6550365Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:36.6551133Z Traceback (most recent call last): 2022-08-17T13:00:36.6551640Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6552001Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6552529Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:36.6552832Z rref_type = type(rref.local_value()) 2022-08-17T13:00:36.6553250Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:36.6553809Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:36.6554152Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:36.6554421Z ValueError('Expected error') 2022-08-17T13:00:36.6554644Z Traceback (most recent call last): 2022-08-17T13:00:36.6555093Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:36.6560615Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:36.6561621Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:36.6562218Z raise ValueError(expected_err) 2022-08-17T13:00:36.6562569Z ValueError: Expected error 2022-08-17T13:00:36.6562699Z 2022-08-17T13:00:36.6562704Z 2022-08-17T13:00:37.0620441Z ok (2.574s) 2022-08-17T13:00:37.0620668Z 2022-08-17T13:00:37.0621063Z ---------------------------------------------------------------------- 2022-08-17T13:00:37.0621329Z Ran 1 test in 2.574s 2022-08-17T13:00:37.0621455Z 2022-08-17T13:00:37.0621519Z OK 2022-08-17T13:00:37.0621615Z 2022-08-17T13:00:37.0621712Z Generating XML reports... 2022-08-17T13:00:37.0656658Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130034.xml 2022-08-17T13:00:38.3834940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:38.3835327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:38.3838235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:38.3838621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:38.5384042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgixk_fi 2022-08-17T13:00:38.5387546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgixk_fi/_remote_module_non_scriptable.py 2022-08-17T13:00:38.8072250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:38.8083373Z 2022-08-17T13:00:38.8083488Z Running tests... 2022-08-17T13:00:38.8084107Z ---------------------------------------------------------------------- 2022-08-17T13:00:39.0825357Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6462 2022-08-17T13:00:39.0855483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6463 2022-08-17T13:00:39.0885808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6464 2022-08-17T13:00:39.0919339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6465 2022-08-17T13:00:40.1312874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:40.1313546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:40.1315087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:40.1315687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:40.1622870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:40.1623531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:40.1625971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:40.1626599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:40.1662830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:40.1663469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:40.1667725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:40.1668379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:40.1707847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:40.1708543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:40.1712034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:40.1712713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:40.2921751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxmxf1rl 2022-08-17T13:00:40.2922739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxmxf1rl/_remote_module_non_scriptable.py 2022-08-17T13:00:40.3230728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9u6qq_h 2022-08-17T13:00:40.3232098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9u6qq_h/_remote_module_non_scriptable.py 2022-08-17T13:00:40.3268207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp178pi8 2022-08-17T13:00:40.3269961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp178pi8/_remote_module_non_scriptable.py 2022-08-17T13:00:40.3317177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_itxt634 2022-08-17T13:00:40.3319027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_itxt634/_remote_module_non_scriptable.py 2022-08-17T13:00:40.5585203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:40.5853782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:40.5913286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:40.5953161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:40.8441445Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:40.8442217Z ValueError('Expected error') 2022-08-17T13:00:40.8442740Z Traceback (most recent call last): 2022-08-17T13:00:40.8443718Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8444472Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8446105Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8446836Z raise ValueError(expected_err) 2022-08-17T13:00:40.8447318Z ValueError: Expected error 2022-08-17T13:00:40.8447579Z 2022-08-17T13:00:40.8454996Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:40.8457542Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:40.8459259Z Traceback (most recent call last): 2022-08-17T13:00:40.8460190Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8460968Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8461944Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:40.8462635Z rref_type = type(rref.local_value()) 2022-08-17T13:00:40.8463561Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:40.8464613Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:40.8465281Z ValueError: On WorkerInfo(id=1, name=worker1): 2022-08-17T13:00:40.8465863Z ValueError('Expected error') 2022-08-17T13:00:40.8466358Z Traceback (most recent call last): 2022-08-17T13:00:40.8467242Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8468002Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8469050Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8469742Z raise ValueError(expected_err) 2022-08-17T13:00:40.8470365Z ValueError: Expected error 2022-08-17T13:00:40.8470647Z 2022-08-17T13:00:40.8470657Z 2022-08-17T13:00:40.8637491Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:40.8638277Z ValueError('Expected error') 2022-08-17T13:00:40.8638702Z Traceback (most recent call last): 2022-08-17T13:00:40.8639513Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8640144Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8641341Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8642198Z raise ValueError(expected_err) 2022-08-17T13:00:40.8642555Z ValueError: Expected error 2022-08-17T13:00:40.8642791Z 2022-08-17T13:00:40.8643201Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:40.8645151Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:40.8646522Z Traceback (most recent call last): 2022-08-17T13:00:40.8647014Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8647366Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8647808Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:40.8648235Z rref_type = type(rref.local_value()) 2022-08-17T13:00:40.8648651Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:40.8649106Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:40.8649438Z ValueError: On WorkerInfo(id=0, name=worker0): 2022-08-17T13:00:40.8649747Z ValueError('Expected error') 2022-08-17T13:00:40.8650014Z Traceback (most recent call last): 2022-08-17T13:00:40.8650603Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8651075Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8651697Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8652060Z raise ValueError(expected_err) 2022-08-17T13:00:40.8652265Z ValueError: Expected error 2022-08-17T13:00:40.8652419Z 2022-08-17T13:00:40.8652427Z 2022-08-17T13:00:40.8765246Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:40.8765991Z ValueError('Expected error') 2022-08-17T13:00:40.8766389Z Traceback (most recent call last): 2022-08-17T13:00:40.8767116Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8767703Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8768510Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8769049Z raise ValueError(expected_err) 2022-08-17T13:00:40.8769422Z ValueError: Expected error 2022-08-17T13:00:40.8769646Z 2022-08-17T13:00:40.8769823Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:40.8771575Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:40.8772703Z Traceback (most recent call last): 2022-08-17T13:00:40.8773406Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8773981Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8775030Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:40.8775566Z rref_type = type(rref.local_value()) 2022-08-17T13:00:40.8776332Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:40.8777132Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:40.8777660Z ValueError: On WorkerInfo(id=3, name=worker3): 2022-08-17T13:00:40.8778111Z ValueError('Expected error') 2022-08-17T13:00:40.8778489Z Traceback (most recent call last): 2022-08-17T13:00:40.8779178Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8779741Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8780534Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8781085Z raise ValueError(expected_err) 2022-08-17T13:00:40.8781447Z ValueError: Expected error 2022-08-17T13:00:40.8781672Z 2022-08-17T13:00:40.8781693Z 2022-08-17T13:00:40.8853722Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:40.8854813Z ValueError('Expected error') 2022-08-17T13:00:40.8855740Z Traceback (most recent call last): 2022-08-17T13:00:40.8856909Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8857809Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8858867Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8859557Z raise ValueError(expected_err) 2022-08-17T13:00:40.8860026Z ValueError: Expected error 2022-08-17T13:00:40.8860305Z 2022-08-17T13:00:40.8869762Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:40.8872404Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func\n raise ValueError(expected_err)\nValueError: Expected error\n') 2022-08-17T13:00:40.8874013Z Traceback (most recent call last): 2022-08-17T13:00:40.8874901Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8875668Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8876664Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 431, in _rref_typeof_on_owner 2022-08-17T13:00:40.8877327Z rref_type = type(rref.local_value()) 2022-08-17T13:00:40.8878253Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-08-17T13:00:40.8879267Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-08-17T13:00:40.8879927Z ValueError: On WorkerInfo(id=2, name=worker2): 2022-08-17T13:00:40.8880489Z ValueError('Expected error') 2022-08-17T13:00:40.8880981Z Traceback (most recent call last): 2022-08-17T13:00:40.8881881Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:00:40.8882614Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:00:40.8883641Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:00:40.8884348Z raise ValueError(expected_err) 2022-08-17T13:00:40.8884808Z ValueError: Expected error 2022-08-17T13:00:40.8885087Z 2022-08-17T13:00:40.8885099Z 2022-08-17T13:00:41.2978829Z ok (2.489s) 2022-08-17T13:00:41.2979430Z 2022-08-17T13:00:41.2979896Z ---------------------------------------------------------------------- 2022-08-17T13:00:41.2980317Z Ran 1 test in 2.489s 2022-08-17T13:00:41.2980492Z 2022-08-17T13:00:41.2980589Z OK 2022-08-17T13:00:41.2980748Z 2022-08-17T13:00:41.2980903Z Generating XML reports... 2022-08-17T13:00:41.3015388Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130038.xml 2022-08-17T13:00:42.6147858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:42.6148264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:42.6150598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:42.6150972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:42.7684939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a9ioy58 2022-08-17T13:00:42.7688199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a9ioy58/_remote_module_non_scriptable.py 2022-08-17T13:00:43.0367975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:43.0378903Z 2022-08-17T13:00:43.0379073Z Running tests... 2022-08-17T13:00:43.0379504Z ---------------------------------------------------------------------- 2022-08-17T13:00:43.3097162Z test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6685 2022-08-17T13:00:43.3127790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6686 2022-08-17T13:00:43.3157184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6687 2022-08-17T13:00:43.3191016Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6688 2022-08-17T13:00:44.3420366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:44.3421046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:44.3424603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:44.3425257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:44.3426039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:44.3426595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:44.3430073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:44.3430735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:44.3586504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:44.3587184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:44.3590820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:44.3591540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:44.3677940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:44.3678601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:44.3681629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:44.3682256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:44.5033995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wsp8jdy 2022-08-17T13:00:44.5035081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrvknbz7 2022-08-17T13:00:44.5035799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wsp8jdy/_remote_module_non_scriptable.py 2022-08-17T13:00:44.5036584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrvknbz7/_remote_module_non_scriptable.py 2022-08-17T13:00:44.5176313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xf1wqh9 2022-08-17T13:00:44.5177496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xf1wqh9/_remote_module_non_scriptable.py 2022-08-17T13:00:44.5273692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz15qmj5o 2022-08-17T13:00:44.5274428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz15qmj5o/_remote_module_non_scriptable.py 2022-08-17T13:00:44.7629734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:44.7650138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:44.7770749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:44.7890525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:45.4249934Z ok (2.387s) 2022-08-17T13:00:45.4250136Z 2022-08-17T13:00:45.4250531Z ---------------------------------------------------------------------- 2022-08-17T13:00:45.4250795Z Ran 1 test in 2.387s 2022-08-17T13:00:45.4250921Z 2022-08-17T13:00:45.4250985Z OK 2022-08-17T13:00:45.4251079Z 2022-08-17T13:00:45.4251159Z Generating XML reports... 2022-08-17T13:00:45.4286543Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130043.xml 2022-08-17T13:00:46.7395475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:46.7395856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:46.7398789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:46.7399194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:46.8959256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnem1eqa 2022-08-17T13:00:46.8962487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnem1eqa/_remote_module_non_scriptable.py 2022-08-17T13:00:47.1658300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:47.1668484Z 2022-08-17T13:00:47.1668754Z Running tests... 2022-08-17T13:00:47.1669382Z ---------------------------------------------------------------------- 2022-08-17T13:00:47.4456627Z test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6908 2022-08-17T13:00:47.4486140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6909 2022-08-17T13:00:47.4519441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6910 2022-08-17T13:00:47.4550010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6911 2022-08-17T13:00:48.4493628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:48.4494316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:48.4497667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:48.4498642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:48.4779282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:48.4780005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:48.4782336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:48.4783020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:48.5078297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:48.5078865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:48.5081680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:48.5082330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:48.5146774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:48.5147462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:48.5151064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:48.5151676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:48.6156046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9siup8nl 2022-08-17T13:00:48.6156856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9siup8nl/_remote_module_non_scriptable.py 2022-08-17T13:00:48.6407649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu9pfuf7 2022-08-17T13:00:48.6408713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu9pfuf7/_remote_module_non_scriptable.py 2022-08-17T13:00:48.6697520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys7g_wqa 2022-08-17T13:00:48.6698343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys7g_wqa/_remote_module_non_scriptable.py 2022-08-17T13:00:48.6759531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95xzy_yg 2022-08-17T13:00:48.6760901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95xzy_yg/_remote_module_non_scriptable.py 2022-08-17T13:00:48.8828196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:48.9157233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:48.9375196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:48.9417297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:49.6610577Z ok (2.494s) 2022-08-17T13:00:49.6610942Z 2022-08-17T13:00:49.6611411Z ---------------------------------------------------------------------- 2022-08-17T13:00:49.6611749Z Ran 1 test in 2.494s 2022-08-17T13:00:49.6611909Z 2022-08-17T13:00:49.6611974Z OK 2022-08-17T13:00:49.6612057Z 2022-08-17T13:00:49.6612167Z Generating XML reports... 2022-08-17T13:00:49.6644471Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130047.xml 2022-08-17T13:00:50.9715872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:50.9716316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:50.9719113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:50.9719490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:51.1263139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgia62q53 2022-08-17T13:00:51.1266466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgia62q53/_remote_module_non_scriptable.py 2022-08-17T13:00:51.3915251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:51.3926551Z 2022-08-17T13:00:51.3926971Z Running tests... 2022-08-17T13:00:51.3927377Z ---------------------------------------------------------------------- 2022-08-17T13:00:51.6617735Z test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7131 2022-08-17T13:00:51.6651217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7132 2022-08-17T13:00:51.6681179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7133 2022-08-17T13:00:51.6711515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7134 2022-08-17T13:00:52.6944388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:52.6945024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:52.6948248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:52.6948967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:52.6973376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:52.6974245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:52.6978888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:52.6979545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:52.6993548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:52.6994164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:52.6998365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:52.6999052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:52.7038663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:52.7039298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:52.7042813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:52.7043488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:52.8546683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhyqiu42 2022-08-17T13:00:52.8547530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhyqiu42/_remote_module_non_scriptable.py 2022-08-17T13:00:52.8589171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfczht83 2022-08-17T13:00:52.8590721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfczht83/_remote_module_non_scriptable.py 2022-08-17T13:00:52.8607799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxj8u7dg 2022-08-17T13:00:52.8610637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxj8u7dg/_remote_module_non_scriptable.py 2022-08-17T13:00:52.8633841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7x_rmrc 2022-08-17T13:00:52.8635807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7x_rmrc/_remote_module_non_scriptable.py 2022-08-17T13:00:53.1162237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:53.1244185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:53.1258287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:53.1264202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:53.7771357Z ok (2.384s) 2022-08-17T13:00:53.7771538Z 2022-08-17T13:00:53.7771933Z ---------------------------------------------------------------------- 2022-08-17T13:00:53.7772183Z Ran 1 test in 2.384s 2022-08-17T13:00:53.7772309Z 2022-08-17T13:00:53.7772372Z OK 2022-08-17T13:00:53.7772488Z 2022-08-17T13:00:53.7772582Z Generating XML reports... 2022-08-17T13:00:53.7809228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130051.xml 2022-08-17T13:00:55.0984988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:55.0985402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:55.0987717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:55.0988093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:55.2505728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3l6b2var 2022-08-17T13:00:55.2508793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3l6b2var/_remote_module_non_scriptable.py 2022-08-17T13:00:55.5140277Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:55.5150614Z 2022-08-17T13:00:55.5150810Z Running tests... 2022-08-17T13:00:55.5151154Z ---------------------------------------------------------------------- 2022-08-17T13:00:55.7834435Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7354 2022-08-17T13:00:55.7862422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7355 2022-08-17T13:00:55.7893759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7356 2022-08-17T13:00:55.7923029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7357 2022-08-17T13:00:56.8569259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:56.8569870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:56.8572271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:56.8572947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:56.8868399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:56.8869045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:56.8872312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:56.8873007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:56.8909985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:56.8910608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:56.8915036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:56.8915638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:56.9297837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:56.9298502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:56.9301246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:56.9301919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:57.0165284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a1azg0i 2022-08-17T13:00:57.0166610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a1azg0i/_remote_module_non_scriptable.py 2022-08-17T13:00:57.0486885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwhswakc 2022-08-17T13:00:57.0488379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwhswakc/_remote_module_non_scriptable.py 2022-08-17T13:00:57.0536371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0mofhbb 2022-08-17T13:00:57.0537992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0mofhbb/_remote_module_non_scriptable.py 2022-08-17T13:00:57.0902220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8jzilm0 2022-08-17T13:00:57.0903012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8jzilm0/_remote_module_non_scriptable.py 2022-08-17T13:00:57.2792701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:00:57.3140457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:00:57.3210031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:00:57.3550976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:00:58.0985875Z ok (2.583s) 2022-08-17T13:00:58.0986111Z 2022-08-17T13:00:58.0986560Z ---------------------------------------------------------------------- 2022-08-17T13:00:58.0986876Z Ran 1 test in 2.583s 2022-08-17T13:00:58.0987028Z 2022-08-17T13:00:58.0987094Z OK 2022-08-17T13:00:58.0987193Z 2022-08-17T13:00:58.0987289Z Generating XML reports... 2022-08-17T13:00:58.1020109Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130055.xml 2022-08-17T13:00:59.4202363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:00:59.4202787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:00:59.4205007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:00:59.4205503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:00:59.5765338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxlxkua9 2022-08-17T13:00:59.5768585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxlxkua9/_remote_module_non_scriptable.py 2022-08-17T13:00:59.8429404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:00:59.8439992Z 2022-08-17T13:00:59.8440424Z Running tests... 2022-08-17T13:00:59.8441026Z ---------------------------------------------------------------------- 2022-08-17T13:01:00.1108281Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7577 2022-08-17T13:01:00.1138629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7578 2022-08-17T13:01:00.1168100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7579 2022-08-17T13:01:00.1200663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7580 2022-08-17T13:01:01.1479017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:01.1479669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:01.1482792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:01.1483380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:01.1660869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:01.1661853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:01.1663611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:01.1664350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:01.1835018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:01.1835669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:01.1838491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:01.1839105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:01.1975322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:01.1976001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:01.1979021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:01.1979954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:01.3091724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv02le4uj 2022-08-17T13:01:01.3092573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv02le4uj/_remote_module_non_scriptable.py 2022-08-17T13:01:01.3282735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwz2qdcu 2022-08-17T13:01:01.3283502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwz2qdcu/_remote_module_non_scriptable.py 2022-08-17T13:01:01.3466808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7ytrqwf 2022-08-17T13:01:01.3467996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7ytrqwf/_remote_module_non_scriptable.py 2022-08-17T13:01:01.3584419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22y8lrmv 2022-08-17T13:01:01.3585119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22y8lrmv/_remote_module_non_scriptable.py 2022-08-17T13:01:01.5730816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:01.5913047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:01.6161323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:01.6236270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:02.3261097Z ok (2.482s) 2022-08-17T13:01:02.3261322Z 2022-08-17T13:01:02.3261666Z ---------------------------------------------------------------------- 2022-08-17T13:01:02.3261953Z Ran 1 test in 2.482s 2022-08-17T13:01:02.3262074Z 2022-08-17T13:01:02.3262124Z OK 2022-08-17T13:01:02.3262221Z 2022-08-17T13:01:02.3262315Z Generating XML reports... 2022-08-17T13:01:02.3295842Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130059.xml 2022-08-17T13:01:03.6449381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:03.6449771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:03.6452466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:03.6452890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:03.7976190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iszs3u5 2022-08-17T13:01:03.7979734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iszs3u5/_remote_module_non_scriptable.py 2022-08-17T13:01:04.0638156Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:04.0649205Z 2022-08-17T13:01:04.0649308Z Running tests... 2022-08-17T13:01:04.0650290Z ---------------------------------------------------------------------- 2022-08-17T13:01:04.3362565Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7800 2022-08-17T13:01:04.3392210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7801 2022-08-17T13:01:04.3423586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7802 2022-08-17T13:01:04.3458363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7803 2022-08-17T13:01:05.4314667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:05.4315375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:05.4317961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:05.4318610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:05.4516262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:05.4516920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:05.4518127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:05.4518731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:05.4796561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:05.4797133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:05.4799942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:05.4800525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:05.4921546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:05.4922098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:05.4924585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:05.4925173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:05.5930300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx04xgnhz 2022-08-17T13:01:05.5931804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx04xgnhz/_remote_module_non_scriptable.py 2022-08-17T13:01:05.6133143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2bdtq6y 2022-08-17T13:01:05.6134001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2bdtq6y/_remote_module_non_scriptable.py 2022-08-17T13:01:05.6427527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_dwnxa9 2022-08-17T13:01:05.6428369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_dwnxa9/_remote_module_non_scriptable.py 2022-08-17T13:01:05.6536442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapeftjj8 2022-08-17T13:01:05.6537549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapeftjj8/_remote_module_non_scriptable.py 2022-08-17T13:01:05.8627641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:05.8837351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:05.9139657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:05.9234683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:06.5517926Z ok (2.487s) 2022-08-17T13:01:06.5518197Z 2022-08-17T13:01:06.5518707Z ---------------------------------------------------------------------- 2022-08-17T13:01:06.5519049Z Ran 1 test in 2.487s 2022-08-17T13:01:06.5519172Z 2022-08-17T13:01:06.5519237Z OK 2022-08-17T13:01:06.5519333Z 2022-08-17T13:01:06.5519428Z Generating XML reports... 2022-08-17T13:01:06.5552848Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130104.xml 2022-08-17T13:01:07.9196018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:07.9196488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:07.9199329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:07.9199815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:08.0755093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgksn7wvx 2022-08-17T13:01:08.0758542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgksn7wvx/_remote_module_non_scriptable.py 2022-08-17T13:01:08.3635505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:08.3646907Z 2022-08-17T13:01:08.3647055Z Running tests... 2022-08-17T13:01:08.3647456Z ---------------------------------------------------------------------- 2022-08-17T13:01:08.6334835Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8023 2022-08-17T13:01:08.6364288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8024 2022-08-17T13:01:08.6395036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8025 2022-08-17T13:01:08.6429942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8026 2022-08-17T13:01:09.6587553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:09.6588237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:09.6589167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:09.6589775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:09.6590572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:09.6591190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:09.6592023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:09.6592639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:09.6789928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:09.6790604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:09.6792199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:09.6792838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:09.7002096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:09.7002716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:09.7005520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:09.7006817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:09.8203398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprksxggjq 2022-08-17T13:01:09.8204130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6732l5q 2022-08-17T13:01:09.8204788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprksxggjq/_remote_module_non_scriptable.py 2022-08-17T13:01:09.8205475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6732l5q/_remote_module_non_scriptable.py 2022-08-17T13:01:09.8383194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmwzl9h3 2022-08-17T13:01:09.8384596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmwzl9h3/_remote_module_non_scriptable.py 2022-08-17T13:01:09.8607463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxakm_et1 2022-08-17T13:01:09.8608335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxakm_et1/_remote_module_non_scriptable.py 2022-08-17T13:01:10.0820961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:10.0830684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:10.1008861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:10.1208958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:10.7487575Z ok (2.384s) 2022-08-17T13:01:10.7487832Z 2022-08-17T13:01:10.7488187Z ---------------------------------------------------------------------- 2022-08-17T13:01:10.7488442Z Ran 1 test in 2.384s 2022-08-17T13:01:10.7488568Z 2022-08-17T13:01:10.7488636Z OK 2022-08-17T13:01:10.7488733Z 2022-08-17T13:01:10.7488828Z Generating XML reports... 2022-08-17T13:01:10.7521774Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130108.xml 2022-08-17T13:01:12.0656894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:12.0657310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:12.0659600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:12.0659978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:12.2206047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcy_474gg 2022-08-17T13:01:12.2209283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy_474gg/_remote_module_non_scriptable.py 2022-08-17T13:01:12.4839311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:12.4849860Z 2022-08-17T13:01:12.4850160Z Running tests... 2022-08-17T13:01:12.4850770Z ---------------------------------------------------------------------- 2022-08-17T13:01:12.7546808Z test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8246 2022-08-17T13:01:12.7575224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8247 2022-08-17T13:01:12.7603878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8248 2022-08-17T13:01:12.7636470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8249 2022-08-17T13:01:13.7575750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:13.7576383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:13.7578686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:13.7579247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:13.8109416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:13.8109974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:13.8113488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:13.8113998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:13.8297569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:13.8298311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:13.8301264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:13.8301986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:13.8441928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:13.8442547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:13.8446860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:13.8447526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:13.9183282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6c058dzi 2022-08-17T13:01:13.9183972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6c058dzi/_remote_module_non_scriptable.py 2022-08-17T13:01:13.9725996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv5cafh2 2022-08-17T13:01:13.9726885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv5cafh2/_remote_module_non_scriptable.py 2022-08-17T13:01:13.9930770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdn487gl 2022-08-17T13:01:13.9931851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdn487gl/_remote_module_non_scriptable.py 2022-08-17T13:01:14.0043444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxevkremq 2022-08-17T13:01:14.0044211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxevkremq/_remote_module_non_scriptable.py 2022-08-17T13:01:14.1822370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:14.2367436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:14.2622577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:14.2697969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:14.9695770Z ok (2.484s) 2022-08-17T13:01:14.9696080Z 2022-08-17T13:01:14.9696598Z ---------------------------------------------------------------------- 2022-08-17T13:01:14.9697070Z Ran 1 test in 2.485s 2022-08-17T13:01:14.9697194Z 2022-08-17T13:01:14.9697259Z OK 2022-08-17T13:01:14.9697345Z 2022-08-17T13:01:14.9697441Z Generating XML reports... 2022-08-17T13:01:14.9734154Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130112.xml 2022-08-17T13:01:16.2727458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:16.2727845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:16.2730136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:16.2730683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:16.4251546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrp_06dd 2022-08-17T13:01:16.4255015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrp_06dd/_remote_module_non_scriptable.py 2022-08-17T13:01:16.6905866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:16.6915528Z 2022-08-17T13:01:16.6915664Z Running tests... 2022-08-17T13:01:16.6916114Z ---------------------------------------------------------------------- 2022-08-17T13:01:16.9596487Z test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8469 2022-08-17T13:01:16.9625573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8470 2022-08-17T13:01:16.9654749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8471 2022-08-17T13:01:16.9689051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8472 2022-08-17T13:01:17.9553795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:17.9554383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:17.9557326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:17.9558239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:18.0173428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:18.0174083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:18.0174745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:18.0175249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:18.0175920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:18.0176450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:18.0177116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:18.0177649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:18.0322130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:18.0322921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:18.0324966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:18.0325704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:18.1140324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1s_ogy7 2022-08-17T13:01:18.1141132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1s_ogy7/_remote_module_non_scriptable.py 2022-08-17T13:01:18.1784159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvuie9dy 2022-08-17T13:01:18.1785004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvuie9dy/_remote_module_non_scriptable.py 2022-08-17T13:01:18.1791843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3s6tlq3 2022-08-17T13:01:18.1794448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3s6tlq3/_remote_module_non_scriptable.py 2022-08-17T13:01:18.1916806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3jd9yp2 2022-08-17T13:01:18.1917925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3jd9yp2/_remote_module_non_scriptable.py 2022-08-17T13:01:18.3787779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:18.4461380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:18.4468632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:18.4571479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:19.0747270Z ok (2.383s) 2022-08-17T13:01:19.0747767Z 2022-08-17T13:01:19.0748205Z ---------------------------------------------------------------------- 2022-08-17T13:01:19.0748463Z Ran 1 test in 2.383s 2022-08-17T13:01:19.0748585Z 2022-08-17T13:01:19.0748651Z OK 2022-08-17T13:01:19.0748748Z 2022-08-17T13:01:19.0748845Z Generating XML reports... 2022-08-17T13:01:19.0782086Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130116.xml 2022-08-17T13:01:20.4123638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:20.4124046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:20.4126441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:20.4126937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:20.5642144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuggnggpp 2022-08-17T13:01:20.5645104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuggnggpp/_remote_module_non_scriptable.py 2022-08-17T13:01:20.8334386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:20.8345667Z 2022-08-17T13:01:20.8345818Z Running tests... 2022-08-17T13:01:20.8346439Z ---------------------------------------------------------------------- 2022-08-17T13:01:21.1042544Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8692 2022-08-17T13:01:21.1072114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8693 2022-08-17T13:01:21.1101107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8694 2022-08-17T13:01:21.1133385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8695 2022-08-17T13:01:22.0989706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:22.0990358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:22.0993234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:22.0993873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:22.1162553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:22.1163196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:22.1166860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:22.1167556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:22.1442293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:22.1442969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:22.1445245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:22.1446049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:22.1738381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:22.1739029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:22.1742617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:22.1743323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:22.2583052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27e9o1ws 2022-08-17T13:01:22.2584182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27e9o1ws/_remote_module_non_scriptable.py 2022-08-17T13:01:22.2768436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffnv24oi 2022-08-17T13:01:22.2769185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffnv24oi/_remote_module_non_scriptable.py 2022-08-17T13:01:22.3013893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fz3ba3h 2022-08-17T13:01:22.3015800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fz3ba3h/_remote_module_non_scriptable.py 2022-08-17T13:01:22.3315635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnszol4ai 2022-08-17T13:01:22.3316952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnszol4ai/_remote_module_non_scriptable.py 2022-08-17T13:01:22.5188011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:22.5398517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:22.5595628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:22.5935571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:23.2192598Z ok (2.384s) 2022-08-17T13:01:23.2192849Z 2022-08-17T13:01:23.2193258Z ---------------------------------------------------------------------- 2022-08-17T13:01:23.2193544Z Ran 1 test in 2.385s 2022-08-17T13:01:23.2193669Z 2022-08-17T13:01:23.2193736Z OK 2022-08-17T13:01:23.2193847Z 2022-08-17T13:01:23.2193930Z Generating XML reports... 2022-08-17T13:01:23.2226939Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130120.xml 2022-08-17T13:01:24.5022794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:24.5023507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:24.5026528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:24.5027120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:24.6524714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaz8n4tz 2022-08-17T13:01:24.6528707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaz8n4tz/_remote_module_non_scriptable.py 2022-08-17T13:01:24.9150405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:24.9160738Z 2022-08-17T13:01:24.9160820Z Running tests... 2022-08-17T13:01:24.9161405Z ---------------------------------------------------------------------- 2022-08-17T13:01:25.1802192Z test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8915 2022-08-17T13:01:25.1830391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8916 2022-08-17T13:01:25.1862828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8917 2022-08-17T13:01:25.1893618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8918 2022-08-17T13:01:26.1848828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:26.1849453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:26.1852728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:26.1853724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:26.1916964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:26.1917607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:26.1921304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:26.1921976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:26.2044582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:26.2045217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:26.2049297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:26.2049973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:26.2501089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:26.2502110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:26.2505227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:26.2506293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:26.3505357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56e3ri17 2022-08-17T13:01:26.3506191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56e3ri17/_remote_module_non_scriptable.py 2022-08-17T13:01:26.3559985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw59ehk2l 2022-08-17T13:01:26.3561573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw59ehk2l/_remote_module_non_scriptable.py 2022-08-17T13:01:26.3696619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ajeg61b 2022-08-17T13:01:26.3698711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ajeg61b/_remote_module_non_scriptable.py 2022-08-17T13:01:26.4087131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwoq0lba 2022-08-17T13:01:26.4088186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwoq0lba/_remote_module_non_scriptable.py 2022-08-17T13:01:26.6121518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:26.6214575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:26.6342359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:26.6728006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:26.6909296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:01:26.7100648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:01:26.7201012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:01:26.7201722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:01:26.7202814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:01:26.7203379Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:01:26.7203929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:01:26.7212519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:01:26.9448121Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:01:26.9450593Z RuntimeError('simulation') 2022-08-17T13:01:26.9450989Z Traceback (most recent call last): 2022-08-17T13:01:26.9451915Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:01:26.9452645Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:01:26.9453502Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-08-17T13:01:26.9454084Z raise RuntimeError('simulation') 2022-08-17T13:01:26.9454393Z RuntimeError: simulation 2022-08-17T13:01:26.9454690Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:01:26.9455070Z RuntimeError('simulation') 2022-08-17T13:01:26.9455381Z Traceback (most recent call last): 2022-08-17T13:01:26.9456027Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:01:26.9456570Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:01:26.9458491Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-08-17T13:01:26.9459118Z raise RuntimeError('simulation') 2022-08-17T13:01:26.9459410Z RuntimeError: simulation 2022-08-17T13:01:26.9460017Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:01:26.9460599Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:01:26.9460993Z RuntimeError('simulation') 2022-08-17T13:01:26.9461337Z Traceback (most recent call last): 2022-08-17T13:01:26.9462017Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:01:26.9462601Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:01:26.9463340Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4523, in raise_error 2022-08-17T13:01:26.9463922Z raise RuntimeError('simulation') 2022-08-17T13:01:26.9464350Z RuntimeError: simulation 2022-08-17T13:01:26.9464532Z 2022-08-17T13:01:26.9464542Z 2022-08-17T13:01:26.9465016Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #4 to worker0, as the agent is shutting down 2022-08-17T13:01:26.9465394Z 2022-08-17T13:01:26.9465689Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:26.9466543Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:26.9467689Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down 2022-08-17T13:01:26.9468322Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:01:26.9469297Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:01:26.9470059Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:26.9470735Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:26.9476012Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:26.9483625Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker3', 'worker1', 'worker2'] timed out in _all_gather after 0.00 seconds. The first exception is eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:01:27.3952267Z ok (2.479s) 2022-08-17T13:01:27.3952485Z 2022-08-17T13:01:27.3953011Z ---------------------------------------------------------------------- 2022-08-17T13:01:27.3953446Z Ran 1 test in 2.479s 2022-08-17T13:01:27.3953567Z 2022-08-17T13:01:27.3953622Z OK 2022-08-17T13:01:27.3953718Z 2022-08-17T13:01:27.3953810Z Generating XML reports... 2022-08-17T13:01:27.3990174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130124.xml 2022-08-17T13:01:28.7201504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:28.7202180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:28.7204452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:28.7204855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:28.8770183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01u29b81 2022-08-17T13:01:28.8773714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01u29b81/_remote_module_non_scriptable.py 2022-08-17T13:01:29.1482982Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:29.1493941Z 2022-08-17T13:01:29.1494046Z Running tests... 2022-08-17T13:01:29.1494525Z ---------------------------------------------------------------------- 2022-08-17T13:01:29.4239301Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9150 2022-08-17T13:01:29.4268753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9151 2022-08-17T13:01:29.4298337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9152 2022-08-17T13:01:29.4329079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9153 2022-08-17T13:01:30.4433611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:30.4434244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:30.4437261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:30.4437907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:30.5132098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:30.5132785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:30.5135582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:30.5136261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:30.5164374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:30.5164958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:30.5169148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:30.5169807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:30.5408908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:30.5409535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:30.5412783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:30.5413454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:30.6027894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_1i_fwf 2022-08-17T13:01:30.6028704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_1i_fwf/_remote_module_non_scriptable.py 2022-08-17T13:01:30.6741532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmn8pfuj 2022-08-17T13:01:30.6742503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmn8pfuj/_remote_module_non_scriptable.py 2022-08-17T13:01:30.6768452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9olv8ivn 2022-08-17T13:01:30.6770853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9olv8ivn/_remote_module_non_scriptable.py 2022-08-17T13:01:30.7019331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmchz6ndm 2022-08-17T13:01:30.7020752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmchz6ndm/_remote_module_non_scriptable.py 2022-08-17T13:01:30.8654931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:30.9367175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:30.9397576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:30.9679227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:31.6388922Z ok (2.489s) 2022-08-17T13:01:31.6389166Z 2022-08-17T13:01:31.6389654Z ---------------------------------------------------------------------- 2022-08-17T13:01:31.6390129Z Ran 1 test in 2.489s 2022-08-17T13:01:31.6390349Z 2022-08-17T13:01:31.6390468Z OK 2022-08-17T13:01:31.6390581Z 2022-08-17T13:01:31.6390678Z Generating XML reports... 2022-08-17T13:01:31.6424634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130129.xml 2022-08-17T13:01:32.9528220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:32.9528659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:32.9530960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:32.9531370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:33.1084794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgcfkcdj 2022-08-17T13:01:33.1088537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgcfkcdj/_remote_module_non_scriptable.py 2022-08-17T13:01:33.3749283Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:33.3759718Z 2022-08-17T13:01:33.3760049Z Running tests... 2022-08-17T13:01:33.3760470Z ---------------------------------------------------------------------- 2022-08-17T13:01:33.6438989Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9373 2022-08-17T13:01:33.6467452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9374 2022-08-17T13:01:33.6496116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9375 2022-08-17T13:01:33.6527549Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9376 2022-08-17T13:01:34.7023526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:34.7024245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:34.7027136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:34.7027730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:34.7094455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:34.7095141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:34.7099256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:34.7099907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:34.7255036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:34.7255707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:34.7257828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:34.7258748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:34.7299105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:34.7299742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:34.7303201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:34.7303867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:34.8622412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpff16l4mh 2022-08-17T13:01:34.8623257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff16l4mh/_remote_module_non_scriptable.py 2022-08-17T13:01:34.8697204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotukk7vw 2022-08-17T13:01:34.8698680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotukk7vw/_remote_module_non_scriptable.py 2022-08-17T13:01:34.8877456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7u5zky3 2022-08-17T13:01:34.8878396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7u5zky3/_remote_module_non_scriptable.py 2022-08-17T13:01:34.8887400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz8rpr71 2022-08-17T13:01:34.8890235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz8rpr71/_remote_module_non_scriptable.py 2022-08-17T13:01:35.1224731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:35.1303593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:35.1502961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:35.1507686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:35.4246491Z Rank 0 finished testing 20 times in 0.07783126831054688 seconds. 2022-08-17T13:01:35.4280963Z Rank 3 finished testing 20 times in 0.06137394905090332 seconds. 2022-08-17T13:01:35.4329687Z Rank 2 finished testing 20 times in 0.0662839412689209 seconds. 2022-08-17T13:01:35.4666380Z Rank 1 finished testing 20 times in 0.09200429916381836 seconds. 2022-08-17T13:01:35.8588001Z ok (2.482s) 2022-08-17T13:01:35.8588255Z 2022-08-17T13:01:35.8588806Z ---------------------------------------------------------------------- 2022-08-17T13:01:35.8589250Z Ran 1 test in 2.483s 2022-08-17T13:01:35.8589467Z 2022-08-17T13:01:35.8589579Z OK 2022-08-17T13:01:35.8590079Z 2022-08-17T13:01:35.8590212Z Generating XML reports... 2022-08-17T13:01:35.8622137Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130133.xml 2022-08-17T13:01:37.1617335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:37.1617729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:37.1619537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:37.1620075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:37.3108058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9q4_ah7 2022-08-17T13:01:37.3111366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9q4_ah7/_remote_module_non_scriptable.py 2022-08-17T13:01:37.5728507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:37.5739854Z 2022-08-17T13:01:37.5740185Z Running tests... 2022-08-17T13:01:37.5740834Z ---------------------------------------------------------------------- 2022-08-17T13:01:37.8367975Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9596 2022-08-17T13:01:37.8395570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9597 2022-08-17T13:01:37.8424844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9598 2022-08-17T13:01:37.8456880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9599 2022-08-17T13:01:38.9451085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:38.9451689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:38.9454271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:38.9454941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:38.9497378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:38.9497984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:38.9501416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:38.9502095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:38.9734809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:38.9735384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:38.9737992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:38.9738601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:38.9750161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:38.9750805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:38.9754923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:38.9755780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:39.1051992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj9mihq3 2022-08-17T13:01:39.1052905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj9mihq3/_remote_module_non_scriptable.py 2022-08-17T13:01:39.1100018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugntvhe3 2022-08-17T13:01:39.1101993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugntvhe3/_remote_module_non_scriptable.py 2022-08-17T13:01:39.1335186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvy_zy7p 2022-08-17T13:01:39.1336576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvy_zy7p/_remote_module_non_scriptable.py 2022-08-17T13:01:39.1355020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_13bit29 2022-08-17T13:01:39.1357764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_13bit29/_remote_module_non_scriptable.py 2022-08-17T13:01:39.3660814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:39.3712976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:39.4022505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:39.4059284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:39.7350248Z Rank 0 finished testing 20 times in 0.1121826171875 seconds. 2022-08-17T13:01:39.7458381Z Rank 1 finished testing 20 times in 0.11309933662414551 seconds. 2022-08-17T13:01:39.7567766Z Rank 3 finished testing 20 times in 0.11391520500183105 seconds. 2022-08-17T13:01:39.7641886Z Rank 2 finished testing 20 times in 0.12090587615966797 seconds. 2022-08-17T13:01:40.1518297Z ok (2.577s) 2022-08-17T13:01:40.1518489Z 2022-08-17T13:01:40.1518812Z ---------------------------------------------------------------------- 2022-08-17T13:01:40.1519079Z Ran 1 test in 2.578s 2022-08-17T13:01:40.1519206Z 2022-08-17T13:01:40.1519271Z OK 2022-08-17T13:01:40.1519352Z 2022-08-17T13:01:40.1519447Z Generating XML reports... 2022-08-17T13:01:40.1551861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130137.xml 2022-08-17T13:01:41.4719520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:41.4719909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:41.4722341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:41.4722723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:41.6258378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m7rsalz 2022-08-17T13:01:41.6262036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m7rsalz/_remote_module_non_scriptable.py 2022-08-17T13:01:41.8946903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:41.8957928Z 2022-08-17T13:01:41.8958561Z Running tests... 2022-08-17T13:01:41.8959096Z ---------------------------------------------------------------------- 2022-08-17T13:01:42.1646572Z test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9819 2022-08-17T13:01:42.1675925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9820 2022-08-17T13:01:42.1705104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9821 2022-08-17T13:01:42.1738925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9822 2022-08-17T13:01:43.1583273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:43.1584028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:43.1587668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:43.1588356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:43.2061978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:43.2062853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:43.2065590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:43.2066175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:43.3178883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73pzilqo 2022-08-17T13:01:43.3179648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73pzilqo/_remote_module_non_scriptable.py 2022-08-17T13:01:43.3263871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:43.3264664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:43.3268039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:43.3268735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:43.3444877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:43.3446051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:43.3448941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:43.3449582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:43.3653195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkp_y6j9 2022-08-17T13:01:43.3653959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkp_y6j9/_remote_module_non_scriptable.py 2022-08-17T13:01:43.4869056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6axk_89 2022-08-17T13:01:43.4870247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6axk_89/_remote_module_non_scriptable.py 2022-08-17T13:01:43.5161522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply6x0f6_ 2022-08-17T13:01:43.5162326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply6x0f6_/_remote_module_non_scriptable.py 2022-08-17T13:01:43.5856107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:43.6282807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:43.7596101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:43.7827181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:44.2034468Z Rank 1 finished testing 1000 times in 0.21275877952575684 seconds. 2022-08-17T13:01:44.2166957Z Rank 3 finished testing 1000 times in 0.2261364459991455 seconds. 2022-08-17T13:01:44.2452038Z Rank 0 finished testing 1000 times in 0.2747986316680908 seconds. 2022-08-17T13:01:44.2824882Z Rank 2 finished testing 1000 times in 0.29189324378967285 seconds. 2022-08-17T13:01:44.6804436Z ok (2.784s) 2022-08-17T13:01:44.6804607Z 2022-08-17T13:01:44.6804960Z ---------------------------------------------------------------------- 2022-08-17T13:01:44.6805222Z Ran 1 test in 2.785s 2022-08-17T13:01:44.6805341Z 2022-08-17T13:01:44.6805405Z OK 2022-08-17T13:01:44.6805501Z 2022-08-17T13:01:44.6805581Z Generating XML reports... 2022-08-17T13:01:44.6838791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130141.xml 2022-08-17T13:01:45.9981810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:45.9982213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:45.9984524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:45.9985165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:46.1552244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw7z591_ 2022-08-17T13:01:46.1555169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw7z591_/_remote_module_non_scriptable.py 2022-08-17T13:01:46.4226440Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:46.4237534Z 2022-08-17T13:01:46.4237633Z Running tests... 2022-08-17T13:01:46.4238224Z ---------------------------------------------------------------------- 2022-08-17T13:01:46.6874054Z test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10042 2022-08-17T13:01:46.6904632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10043 2022-08-17T13:01:46.6934867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10044 2022-08-17T13:01:46.6967864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10045 2022-08-17T13:01:47.6826893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:47.6827582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:47.6828947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:47.6829542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:47.7006652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:47.7007317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:47.7010768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:47.7011454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:47.7064676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:47.7065266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:47.7069226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:47.7069904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:47.7546736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:47.7547249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:47.7550906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:47.7551706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:47.8430543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytohe62m 2022-08-17T13:01:47.8431406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytohe62m/_remote_module_non_scriptable.py 2022-08-17T13:01:47.8624618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnug2mj4u 2022-08-17T13:01:47.8625950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnug2mj4u/_remote_module_non_scriptable.py 2022-08-17T13:01:47.8677058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vai0sia 2022-08-17T13:01:47.8679047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vai0sia/_remote_module_non_scriptable.py 2022-08-17T13:01:47.9134442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7yq1rck 2022-08-17T13:01:47.9135519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7yq1rck/_remote_module_non_scriptable.py 2022-08-17T13:01:48.1063818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:48.1311463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:48.1350284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:48.1969093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:48.5021917Z ok (2.078s) 2022-08-17T13:01:48.5022097Z 2022-08-17T13:01:48.5022512Z ---------------------------------------------------------------------- 2022-08-17T13:01:48.5022796Z Ran 1 test in 2.078s 2022-08-17T13:01:48.5022904Z 2022-08-17T13:01:48.5022967Z OK 2022-08-17T13:01:48.5023066Z 2022-08-17T13:01:48.5023163Z Generating XML reports... 2022-08-17T13:01:48.5056366Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130146.xml 2022-08-17T13:01:49.8195258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:49.8195658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:49.8198860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:49.8199323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:49.9711266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs0yvy_2 2022-08-17T13:01:49.9714280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs0yvy_2/_remote_module_non_scriptable.py 2022-08-17T13:01:50.2356410Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:50.2366922Z 2022-08-17T13:01:50.2367050Z Running tests... 2022-08-17T13:01:50.2367560Z ---------------------------------------------------------------------- 2022-08-17T13:01:50.5021385Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10097 2022-08-17T13:01:50.5049487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10098 2022-08-17T13:01:50.5078721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10099 2022-08-17T13:01:50.5111420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10100 2022-08-17T13:01:51.4996936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:51.4997657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:51.4999690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:51.5000290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:51.5145178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:51.5145852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:51.5149201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:51.5149841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:51.5464797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:51.5465488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:51.5467960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:51.5468621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:51.5566615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:51.5567323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:51.5570712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:51.5571415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:51.6655434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jv3bhhm 2022-08-17T13:01:51.6656816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jv3bhhm/_remote_module_non_scriptable.py 2022-08-17T13:01:51.6799086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qfjq7at 2022-08-17T13:01:51.6801319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qfjq7at/_remote_module_non_scriptable.py 2022-08-17T13:01:51.7068920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqm8h4ho 2022-08-17T13:01:51.7070183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqm8h4ho/_remote_module_non_scriptable.py 2022-08-17T13:01:51.7178539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkolql7lq 2022-08-17T13:01:51.7179614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkolql7lq/_remote_module_non_scriptable.py 2022-08-17T13:01:51.9415309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:51.9512992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:51.9698043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:51.9824003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:52.6171182Z ok (2.380s) 2022-08-17T13:01:52.6171475Z 2022-08-17T13:01:52.6171979Z ---------------------------------------------------------------------- 2022-08-17T13:01:52.6172263Z Ran 1 test in 2.380s 2022-08-17T13:01:52.6172386Z 2022-08-17T13:01:52.6172437Z OK 2022-08-17T13:01:52.6172534Z 2022-08-17T13:01:52.6172628Z Generating XML reports... 2022-08-17T13:01:52.6205171Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130150.xml 2022-08-17T13:01:53.9571616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:53.9572342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:53.9575691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:53.9576282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:54.1126293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lnazvqn 2022-08-17T13:01:54.1130163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lnazvqn/_remote_module_non_scriptable.py 2022-08-17T13:01:54.3804187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:54.3816112Z 2022-08-17T13:01:54.3816388Z Running tests... 2022-08-17T13:01:54.3817019Z ---------------------------------------------------------------------- 2022-08-17T13:01:54.6649770Z test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10320 2022-08-17T13:01:54.6679429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10321 2022-08-17T13:01:54.6708273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10322 2022-08-17T13:01:54.6737846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10323 2022-08-17T13:01:55.6659017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:55.6660046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:55.6660914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:55.6661593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:55.6921771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:55.6922347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:55.6925360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:55.6926297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:55.7311773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:55.7312326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:55.7315047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:55.7315993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:55.7324056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:55.7324715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:55.7329991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:55.7330658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:55.8259789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7_7cyfc 2022-08-17T13:01:55.8260661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7_7cyfc/_remote_module_non_scriptable.py 2022-08-17T13:01:55.8547662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe14w29o 2022-08-17T13:01:55.8548575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe14w29o/_remote_module_non_scriptable.py 2022-08-17T13:01:55.8929560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_sijj4y 2022-08-17T13:01:55.8930542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxetr29yk 2022-08-17T13:01:55.8931292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_sijj4y/_remote_module_non_scriptable.py 2022-08-17T13:01:55.8933670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxetr29yk/_remote_module_non_scriptable.py 2022-08-17T13:01:56.0904763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:01:56.1247924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:01:56.1590917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:01:56.1599801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:01:56.8797965Z ok (2.498s) 2022-08-17T13:01:56.8798259Z 2022-08-17T13:01:56.8798798Z ---------------------------------------------------------------------- 2022-08-17T13:01:56.8799055Z Ran 1 test in 2.498s 2022-08-17T13:01:56.8799177Z 2022-08-17T13:01:56.8799242Z OK 2022-08-17T13:01:56.8799339Z 2022-08-17T13:01:56.8799433Z Generating XML reports... 2022-08-17T13:01:56.8833146Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130154.xml 2022-08-17T13:01:58.1835329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:58.1836286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:58.1838720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:58.1839309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:58.3343499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyvp7451 2022-08-17T13:01:58.3346974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyvp7451/_remote_module_non_scriptable.py 2022-08-17T13:01:58.5955751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:01:58.5965726Z 2022-08-17T13:01:58.5965802Z Running tests... 2022-08-17T13:01:58.5966456Z ---------------------------------------------------------------------- 2022-08-17T13:01:58.8618248Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10563 2022-08-17T13:01:58.8647215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10564 2022-08-17T13:01:58.8675835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10565 2022-08-17T13:01:58.8707312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10566 2022-08-17T13:01:59.8491090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:59.8491807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:59.8493991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:59.8494650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:59.8500274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:59.8500881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:59.8505569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:59.8506436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:59.9046558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:59.9047164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:59.9049145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:59.9049683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:01:59.9209672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:01:59.9210369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:01:59.9214172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:01:59.9214964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:00.0091849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ca2fpuz 2022-08-17T13:02:00.0092657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ca2fpuz/_remote_module_non_scriptable.py 2022-08-17T13:02:00.0099726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceumzmqy 2022-08-17T13:02:00.0101198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceumzmqy/_remote_module_non_scriptable.py 2022-08-17T13:02:00.0628379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzcxfg2d 2022-08-17T13:02:00.0629158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzcxfg2d/_remote_module_non_scriptable.py 2022-08-17T13:02:00.0797475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan27zm0e 2022-08-17T13:02:00.0798244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan27zm0e/_remote_module_non_scriptable.py 2022-08-17T13:02:00.2731457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:00.2738269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:00.3222912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:00.3410830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:01.0767762Z ok (2.480s) 2022-08-17T13:02:01.0767982Z 2022-08-17T13:02:01.0768426Z ---------------------------------------------------------------------- 2022-08-17T13:02:01.0768891Z Ran 1 test in 2.480s 2022-08-17T13:02:01.0769112Z 2022-08-17T13:02:01.0769199Z OK 2022-08-17T13:02:01.0769301Z 2022-08-17T13:02:01.0769417Z Generating XML reports... 2022-08-17T13:02:01.0802935Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130158.xml 2022-08-17T13:02:02.4075652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:02.4076289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:02.4079183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:02.4079583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:02.5609339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cqrgq08 2022-08-17T13:02:02.5612315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cqrgq08/_remote_module_non_scriptable.py 2022-08-17T13:02:02.8293154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:02.8304087Z 2022-08-17T13:02:02.8304482Z Running tests... 2022-08-17T13:02:02.8304898Z ---------------------------------------------------------------------- 2022-08-17T13:02:03.0978597Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10786 2022-08-17T13:02:03.1008340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10787 2022-08-17T13:02:03.1038087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10788 2022-08-17T13:02:03.1070795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10789 2022-08-17T13:02:04.0804963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:04.0805665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:04.0808175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:04.0808838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:04.0995948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:04.0996621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:04.1000069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:04.1000727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:04.1456436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:04.1457129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:04.1460247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:04.1461228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:04.1472375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:04.1472953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:04.1477519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:04.1478226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:04.2391029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxeehd6v5 2022-08-17T13:02:04.2392208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxeehd6v5/_remote_module_non_scriptable.py 2022-08-17T13:02:04.2571296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdvlsx93 2022-08-17T13:02:04.2572499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdvlsx93/_remote_module_non_scriptable.py 2022-08-17T13:02:04.3067328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk3ykbsa 2022-08-17T13:02:04.3068012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10xghk9s 2022-08-17T13:02:04.3068893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk3ykbsa/_remote_module_non_scriptable.py 2022-08-17T13:02:04.3069605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10xghk9s/_remote_module_non_scriptable.py 2022-08-17T13:02:04.5017987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:04.5193764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:04.5722018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:04.5722754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:05.3131053Z ok (2.482s) 2022-08-17T13:02:05.3131271Z 2022-08-17T13:02:05.3131730Z ---------------------------------------------------------------------- 2022-08-17T13:02:05.3132003Z Ran 1 test in 2.483s 2022-08-17T13:02:05.3132127Z 2022-08-17T13:02:05.3132197Z OK 2022-08-17T13:02:05.3132354Z 2022-08-17T13:02:05.3132452Z Generating XML reports... 2022-08-17T13:02:05.3166542Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130202.xml 2022-08-17T13:02:06.6092558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:06.6093231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:06.6096131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:06.6096722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:06.7611853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzauajk_f 2022-08-17T13:02:06.7615656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzauajk_f/_remote_module_non_scriptable.py 2022-08-17T13:02:07.0273591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:07.0285038Z 2022-08-17T13:02:07.0285428Z Running tests... 2022-08-17T13:02:07.0286086Z ---------------------------------------------------------------------- 2022-08-17T13:02:07.3016931Z test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11009 2022-08-17T13:02:07.3046642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11010 2022-08-17T13:02:07.3075402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11011 2022-08-17T13:02:07.3107345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11012 2022-08-17T13:02:08.3982657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:08.3983250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:08.4018240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:08.4019265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:08.4020574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:08.4021225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:08.4021980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:08.4022590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:08.4141856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:08.4142468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:08.4146041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:08.4146692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:08.4857279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:08.4857948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:08.4861262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:08.4861844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:08.5583397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4w5dj6b 2022-08-17T13:02:08.5584377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4w5dj6b/_remote_module_non_scriptable.py 2022-08-17T13:02:08.5603495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvn4ztkl 2022-08-17T13:02:08.5606102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvn4ztkl/_remote_module_non_scriptable.py 2022-08-17T13:02:08.5722642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z5cxolr 2022-08-17T13:02:08.5724091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z5cxolr/_remote_module_non_scriptable.py 2022-08-17T13:02:08.6423433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuag8of23 2022-08-17T13:02:08.6424580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuag8of23/_remote_module_non_scriptable.py 2022-08-17T13:02:08.8171743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:08.8222818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:08.8324199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:08.9034811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:09.5167282Z ok (2.488s) 2022-08-17T13:02:09.5167547Z 2022-08-17T13:02:09.5168087Z ---------------------------------------------------------------------- 2022-08-17T13:02:09.5168550Z Ran 1 test in 2.488s 2022-08-17T13:02:09.5168669Z 2022-08-17T13:02:09.5168733Z OK 2022-08-17T13:02:09.5168830Z 2022-08-17T13:02:09.5168913Z Generating XML reports... 2022-08-17T13:02:09.5203598Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130207.xml 2022-08-17T13:02:10.8165287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:10.8166122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:10.8168378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:10.8168758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:10.9708246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggxgq6ig 2022-08-17T13:02:10.9710995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggxgq6ig/_remote_module_non_scriptable.py 2022-08-17T13:02:11.2337280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:11.2347171Z 2022-08-17T13:02:11.2347268Z Running tests... 2022-08-17T13:02:11.2347774Z ---------------------------------------------------------------------- 2022-08-17T13:02:11.4994768Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11232 2022-08-17T13:02:11.5023793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11233 2022-08-17T13:02:11.5053269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11234 2022-08-17T13:02:11.5083758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11235 2022-08-17T13:02:12.5593636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:12.5594255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:12.5597183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:12.5597882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:12.5735414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:12.5736055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:12.5738576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:12.5739487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:12.6135209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:12.6135816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:12.6138942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:12.6139550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:12.6504715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:12.6505385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:12.6508730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:12.6509515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:12.7191037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzjvou_r 2022-08-17T13:02:12.7192623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzjvou_r/_remote_module_non_scriptable.py 2022-08-17T13:02:12.7333145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppssgfaam 2022-08-17T13:02:12.7334397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppssgfaam/_remote_module_non_scriptable.py 2022-08-17T13:02:12.7706463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpgtp7ww 2022-08-17T13:02:12.7707435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpgtp7ww/_remote_module_non_scriptable.py 2022-08-17T13:02:12.8095150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47ssskjt 2022-08-17T13:02:12.8096657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47ssskjt/_remote_module_non_scriptable.py 2022-08-17T13:02:12.9833459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:12.9982447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:13.0330744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:13.0741886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:13.3042959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:02:13.3146048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:02:13.3147519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:02:13.3148912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:13.3150154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:02:13.3151346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:13.3153919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:13.3156563Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:13.8147107Z ok (2.580s) 2022-08-17T13:02:13.8147342Z 2022-08-17T13:02:13.8147875Z ---------------------------------------------------------------------- 2022-08-17T13:02:13.8148339Z Ran 1 test in 2.580s 2022-08-17T13:02:13.8148554Z 2022-08-17T13:02:13.8148651Z OK 2022-08-17T13:02:13.8148825Z 2022-08-17T13:02:13.8148999Z Generating XML reports... 2022-08-17T13:02:13.8182971Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130211.xml 2022-08-17T13:02:15.1256077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:15.1256473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:15.1259116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:15.1259604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:15.2775968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1src976p 2022-08-17T13:02:15.2779247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1src976p/_remote_module_non_scriptable.py 2022-08-17T13:02:15.5418847Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:15.5430388Z 2022-08-17T13:02:15.5430821Z Running tests... 2022-08-17T13:02:15.5431251Z ---------------------------------------------------------------------- 2022-08-17T13:02:15.8079411Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11467 2022-08-17T13:02:15.8107444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11468 2022-08-17T13:02:15.8136839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11469 2022-08-17T13:02:15.8169848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11470 2022-08-17T13:02:16.8937513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:16.8938457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:16.8941289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:16.8941882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:16.9207388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:16.9208055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:16.9211551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:16.9212196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:16.9535121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:16.9535755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:16.9538595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:16.9539573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:16.9884846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:16.9885485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:16.9888717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:16.9889321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:17.0506556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppbzkten 2022-08-17T13:02:17.0507500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppbzkten/_remote_module_non_scriptable.py 2022-08-17T13:02:17.0789311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc5bld3x 2022-08-17T13:02:17.0790237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc5bld3x/_remote_module_non_scriptable.py 2022-08-17T13:02:17.1152421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoctxb4eg 2022-08-17T13:02:17.1153476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoctxb4eg/_remote_module_non_scriptable.py 2022-08-17T13:02:17.1447218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnjccjcd 2022-08-17T13:02:17.1448166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnjccjcd/_remote_module_non_scriptable.py 2022-08-17T13:02:17.3084610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:17.3404292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:17.3745222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:17.4036343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:17.6455599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:02:17.6558406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:02:17.6558843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:02:17.6561829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:02:17.6564064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:17.6659495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:17.6661388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:17.6666957Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:17.6685650Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:17.6686150Z ValueError('Expected error') 2022-08-17T13:02:17.6686372Z Traceback (most recent call last): 2022-08-17T13:02:17.6686891Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6687287Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6688022Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6688418Z raise ValueError(expected_err) 2022-08-17T13:02:17.6688670Z ValueError: Expected error 2022-08-17T13:02:17.6688995Z 2022-08-17T13:02:17.6691965Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:17.6692425Z ValueError('Expected error') 2022-08-17T13:02:17.6692795Z Traceback (most recent call last): 2022-08-17T13:02:17.6693651Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6694148Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6694736Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6695046Z raise ValueError(expected_err) 2022-08-17T13:02:17.6695267Z ValueError: Expected error 2022-08-17T13:02:17.6695398Z 2022-08-17T13:02:17.6695492Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:17.6695739Z ValueError('Expected error') 2022-08-17T13:02:17.6695962Z Traceback (most recent call last): 2022-08-17T13:02:17.6696379Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6696794Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6699012Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6699370Z raise ValueError(expected_err) 2022-08-17T13:02:17.6699592Z ValueError: Expected error 2022-08-17T13:02:17.6703245Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:17.6703760Z ValueError('Expected error') 2022-08-17T13:02:17.6704166Z Traceback (most recent call last): 2022-08-17T13:02:17.6704616Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6705156Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6705623Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6705962Z raise ValueError(expected_err) 2022-08-17T13:02:17.6706184Z ValueError: Expected error 2022-08-17T13:02:17.6706314Z 2022-08-17T13:02:17.6707179Z 2022-08-17T13:02:17.6813804Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:17.6814532Z ValueError('Expected error') 2022-08-17T13:02:17.6815041Z Traceback (most recent call last): 2022-08-17T13:02:17.6815989Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6816758Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6817706Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6818428Z raise ValueError(expected_err) 2022-08-17T13:02:17.6818911Z ValueError: Expected error 2022-08-17T13:02:17.6821693Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:17.6825173Z ValueError('Expected error') 2022-08-17T13:02:17.6828220Z Traceback (most recent call last): 2022-08-17T13:02:17.6832334Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6847238Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6848200Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6848803Z raise ValueError(expected_err) 2022-08-17T13:02:17.6849180Z ValueError: Expected error 2022-08-17T13:02:17.6850395Z 2022-08-17T13:02:17.6853517Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:17.6853995Z ValueError('Expected error') 2022-08-17T13:02:17.6854404Z Traceback (most recent call last): 2022-08-17T13:02:17.6855151Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6855801Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6856894Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6857491Z raise ValueError(expected_err) 2022-08-17T13:02:17.6857889Z ValueError: Expected error 2022-08-17T13:02:17.6858126Z 2022-08-17T13:02:17.6858302Z 2022-08-17T13:02:17.6858474Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:17.6858929Z ValueError('Expected error') 2022-08-17T13:02:17.6859426Z Traceback (most recent call last): 2022-08-17T13:02:17.6860358Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:17.6861115Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:17.6862155Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:17.6862861Z raise ValueError(expected_err) 2022-08-17T13:02:17.6863331Z ValueError: Expected error 2022-08-17T13:02:17.6863612Z 2022-08-17T13:02:18.1230454Z ok (2.580s) 2022-08-17T13:02:18.1230663Z 2022-08-17T13:02:18.1231175Z ---------------------------------------------------------------------- 2022-08-17T13:02:18.1231642Z Ran 1 test in 2.580s 2022-08-17T13:02:18.1231869Z 2022-08-17T13:02:18.1231978Z OK 2022-08-17T13:02:18.1232117Z 2022-08-17T13:02:18.1232215Z Generating XML reports... 2022-08-17T13:02:18.1265199Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130215.xml 2022-08-17T13:02:19.4403565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:19.4404271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:19.4407086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:19.4407495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:19.5949004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcpu13ok 2022-08-17T13:02:19.5952477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcpu13ok/_remote_module_non_scriptable.py 2022-08-17T13:02:19.8612940Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:19.8622804Z 2022-08-17T13:02:19.8622905Z Running tests... 2022-08-17T13:02:19.8623369Z ---------------------------------------------------------------------- 2022-08-17T13:02:20.1278614Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11702 2022-08-17T13:02:20.1307743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11703 2022-08-17T13:02:20.1337927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11704 2022-08-17T13:02:20.1370987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11705 2022-08-17T13:02:21.1889463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:21.1890060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:21.1892857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:21.1893477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:21.2020320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:21.2020987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:21.2023856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:21.2024518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:21.2471621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:21.2472110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:21.2475081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:21.2475667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:21.2907742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:21.2908403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:21.2910863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:21.2911514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:21.3489837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_b8_gw6 2022-08-17T13:02:21.3490682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_b8_gw6/_remote_module_non_scriptable.py 2022-08-17T13:02:21.3617307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bzg5tym 2022-08-17T13:02:21.3618039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bzg5tym/_remote_module_non_scriptable.py 2022-08-17T13:02:21.4069371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68jhqph9 2022-08-17T13:02:21.4070298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68jhqph9/_remote_module_non_scriptable.py 2022-08-17T13:02:21.4507906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4_uvbx4 2022-08-17T13:02:21.4508827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4_uvbx4/_remote_module_non_scriptable.py 2022-08-17T13:02:21.6123842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:21.6272761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:21.6716850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:21.7185910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:21.9676073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:02:21.9774434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:02:21.9882810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:02:21.9883552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:21.9884346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:21.9884849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:02:21.9885444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:21.9886200Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:22.5434632Z ok (2.681s) 2022-08-17T13:02:22.5434863Z 2022-08-17T13:02:22.5435406Z ---------------------------------------------------------------------- 2022-08-17T13:02:22.5435816Z Ran 1 test in 2.681s 2022-08-17T13:02:22.5435938Z 2022-08-17T13:02:22.5436003Z OK 2022-08-17T13:02:22.5436086Z 2022-08-17T13:02:22.5436179Z Generating XML reports... 2022-08-17T13:02:22.5468156Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130219.xml 2022-08-17T13:02:23.8499178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:23.8499581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:23.8502520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:23.8502911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:24.0020630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllkw5vpo 2022-08-17T13:02:24.0023697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllkw5vpo/_remote_module_non_scriptable.py 2022-08-17T13:02:24.2654523Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:24.2665033Z 2022-08-17T13:02:24.2665155Z Running tests... 2022-08-17T13:02:24.2665798Z ---------------------------------------------------------------------- 2022-08-17T13:02:24.5293816Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11937 2022-08-17T13:02:24.5322578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11938 2022-08-17T13:02:24.5351290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11939 2022-08-17T13:02:24.5382622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11940 2022-08-17T13:02:25.5934636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:25.5935266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:25.5936972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:25.5937625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:25.6025316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:25.6025994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:25.6030167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:25.6030864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:25.6052786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:25.6053399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:25.6057896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:25.6058482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:25.6242112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:25.6242725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:25.6245787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:25.6246661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:25.7535274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcu_q1jz 2022-08-17T13:02:25.7536110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcu_q1jz/_remote_module_non_scriptable.py 2022-08-17T13:02:25.7618848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevhmm98y 2022-08-17T13:02:25.7619671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevhmm98y/_remote_module_non_scriptable.py 2022-08-17T13:02:25.7642475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsg87d5sx 2022-08-17T13:02:25.7645153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsg87d5sx/_remote_module_non_scriptable.py 2022-08-17T13:02:25.7814970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5cm6aac 2022-08-17T13:02:25.7816281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5cm6aac/_remote_module_non_scriptable.py 2022-08-17T13:02:26.0241435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:26.0289059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:26.0418725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:26.0431450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:26.6441594Z ok (2.377s) 2022-08-17T13:02:26.6441871Z 2022-08-17T13:02:26.6442243Z ---------------------------------------------------------------------- 2022-08-17T13:02:26.6442544Z Ran 1 test in 2.378s 2022-08-17T13:02:26.6442673Z 2022-08-17T13:02:26.6442725Z OK 2022-08-17T13:02:26.6442823Z 2022-08-17T13:02:26.6442921Z Generating XML reports... 2022-08-17T13:02:26.6475858Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130224.xml 2022-08-17T13:02:27.9265150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:27.9265828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:27.9268607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:27.9269178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:28.0798915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jwv7jwi 2022-08-17T13:02:28.0802405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jwv7jwi/_remote_module_non_scriptable.py 2022-08-17T13:02:28.3413128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:28.3423907Z 2022-08-17T13:02:28.3424319Z Running tests... 2022-08-17T13:02:28.3424780Z ---------------------------------------------------------------------- 2022-08-17T13:02:28.6016842Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12160 2022-08-17T13:02:28.6045641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12161 2022-08-17T13:02:28.6074756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12162 2022-08-17T13:02:28.6106812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12163 2022-08-17T13:02:29.6667403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:29.6668373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:29.6670006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:29.6670654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:29.6806818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:29.6807528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:29.6810568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:29.6811247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:29.6953329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:29.6953991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:29.6956719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:29.6957655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:29.7047958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:29.7048609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:29.7051622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:29.7052272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:29.8271716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6htoodg 2022-08-17T13:02:29.8272642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6htoodg/_remote_module_non_scriptable.py 2022-08-17T13:02:29.8423246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpin2p2c0n 2022-08-17T13:02:29.8424509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpin2p2c0n/_remote_module_non_scriptable.py 2022-08-17T13:02:29.8548269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei7gdm7o 2022-08-17T13:02:29.8549518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei7gdm7o/_remote_module_non_scriptable.py 2022-08-17T13:02:29.8643572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkon_fz4u 2022-08-17T13:02:29.8644625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkon_fz4u/_remote_module_non_scriptable.py 2022-08-17T13:02:30.1082345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:30.1115359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:30.1194157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:30.1269740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:30.7165320Z ok (2.374s) 2022-08-17T13:02:30.7172107Z 2022-08-17T13:02:30.7173132Z ---------------------------------------------------------------------- 2022-08-17T13:02:30.7173457Z Ran 1 test in 2.374s 2022-08-17T13:02:30.7173585Z 2022-08-17T13:02:30.7173651Z OK 2022-08-17T13:02:30.7173747Z 2022-08-17T13:02:30.7173843Z Generating XML reports... 2022-08-17T13:02:30.7202599Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130228.xml 2022-08-17T13:02:32.0389076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:32.0389759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:32.0392542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:32.0393157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:32.1908129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7uy_bbz 2022-08-17T13:02:32.1911238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7uy_bbz/_remote_module_non_scriptable.py 2022-08-17T13:02:32.4541463Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:32.4551890Z 2022-08-17T13:02:32.4552192Z Running tests... 2022-08-17T13:02:32.4552879Z ---------------------------------------------------------------------- 2022-08-17T13:02:32.7161798Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12383 2022-08-17T13:02:32.7191778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12384 2022-08-17T13:02:32.7221600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12385 2022-08-17T13:02:32.7256507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12386 2022-08-17T13:02:33.7250291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:33.7250995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:33.7252585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:33.7253251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:33.7363525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:33.7364175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:33.7367360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:33.7367976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:33.7611567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:33.7612236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:33.7615091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:33.7615760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:33.7736430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:33.7737073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:33.7740097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:33.7740784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:33.8849432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgm2trv1 2022-08-17T13:02:33.8851077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgm2trv1/_remote_module_non_scriptable.py 2022-08-17T13:02:33.8976672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wu_wr04 2022-08-17T13:02:33.8977402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wu_wr04/_remote_module_non_scriptable.py 2022-08-17T13:02:33.9203655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcc1rad6 2022-08-17T13:02:33.9204541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcc1rad6/_remote_module_non_scriptable.py 2022-08-17T13:02:33.9336428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t1osov7 2022-08-17T13:02:33.9337204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t1osov7/_remote_module_non_scriptable.py 2022-08-17T13:02:34.1454886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:34.1583691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:34.1806755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:34.1927725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:34.4138550Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:34.4139321Z ValueError('Expected error') 2022-08-17T13:02:34.4139819Z Traceback (most recent call last): 2022-08-17T13:02:34.4140754Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:34.4141548Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:34.4142587Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:34.4143288Z raise ValueError(expected_err) 2022-08-17T13:02:34.4144129Z ValueError: Expected error 2022-08-17T13:02:34.4144419Z 2022-08-17T13:02:34.4237408Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:34.4238041Z ValueError('Expected error') 2022-08-17T13:02:34.4238428Z Traceback (most recent call last): 2022-08-17T13:02:34.4239149Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:34.4239720Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:34.4240508Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:34.4241047Z raise ValueError(expected_err) 2022-08-17T13:02:34.4241413Z ValueError: Expected error 2022-08-17T13:02:34.4241629Z 2022-08-17T13:02:34.4320526Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:34.4321126Z ValueError('Expected error') 2022-08-17T13:02:34.4321507Z Traceback (most recent call last): 2022-08-17T13:02:34.4322280Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:34.4322874Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:34.4323665Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:34.4324187Z raise ValueError(expected_err) 2022-08-17T13:02:34.4324559Z ValueError: Expected error 2022-08-17T13:02:34.4324781Z 2022-08-17T13:02:34.4352618Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:34.4353525Z ValueError('Expected error') 2022-08-17T13:02:34.4354180Z Traceback (most recent call last): 2022-08-17T13:02:34.4355267Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:34.4356043Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:34.4357078Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:34.4357794Z raise ValueError(expected_err) 2022-08-17T13:02:34.4358270Z ValueError: Expected error 2022-08-17T13:02:34.4358553Z 2022-08-17T13:02:34.8314414Z ok (2.376s) 2022-08-17T13:02:34.8314655Z 2022-08-17T13:02:34.8315114Z ---------------------------------------------------------------------- 2022-08-17T13:02:34.8315573Z Ran 1 test in 2.376s 2022-08-17T13:02:34.8315790Z 2022-08-17T13:02:34.8315910Z OK 2022-08-17T13:02:34.8316084Z 2022-08-17T13:02:34.8316233Z Generating XML reports... 2022-08-17T13:02:34.8349218Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130232.xml 2022-08-17T13:02:36.1195420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:36.1195893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:36.1197816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:36.1198261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:36.2744356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlr8tvja 2022-08-17T13:02:36.2747822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlr8tvja/_remote_module_non_scriptable.py 2022-08-17T13:02:36.5471828Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:36.5481891Z 2022-08-17T13:02:36.5482399Z Running tests... 2022-08-17T13:02:36.5482824Z ---------------------------------------------------------------------- 2022-08-17T13:02:36.8114980Z test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12606 2022-08-17T13:02:36.8144092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12607 2022-08-17T13:02:36.8173322Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12608 2022-08-17T13:02:36.8204253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12609 2022-08-17T13:02:37.7944801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:37.7945535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:37.7947608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:37.7948209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:37.7949061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:37.7949704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:37.7952193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:37.7952831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:37.7953631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:37.7954256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:37.7957312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:37.7957951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:37.8642977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:37.8643683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:37.8646238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:37.8646908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:37.9551400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e81yxir 2022-08-17T13:02:37.9552151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp27wmvb4 2022-08-17T13:02:37.9552807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e81yxir/_remote_module_non_scriptable.py 2022-08-17T13:02:37.9553339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp27wmvb4/_remote_module_non_scriptable.py 2022-08-17T13:02:37.9554056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav0f3ujf 2022-08-17T13:02:37.9557319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav0f3ujf/_remote_module_non_scriptable.py 2022-08-17T13:02:38.0233456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprywlhqc 2022-08-17T13:02:38.0234368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprywlhqc/_remote_module_non_scriptable.py 2022-08-17T13:02:38.2169464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:38.2192325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:38.2199019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:38.2896502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:38.6628917Z [W tensorpipe_agent.cpp:940] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:39.5212298Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down 2022-08-17T13:02:39.5369007Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down 2022-08-17T13:02:39.5410271Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down 2022-08-17T13:02:39.5467787Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down 2022-08-17T13:02:39.9276914Z ok (3.379s) 2022-08-17T13:02:39.9277134Z 2022-08-17T13:02:39.9277626Z ---------------------------------------------------------------------- 2022-08-17T13:02:39.9277880Z Ran 1 test in 3.379s 2022-08-17T13:02:39.9278007Z 2022-08-17T13:02:39.9278074Z OK 2022-08-17T13:02:39.9278183Z 2022-08-17T13:02:39.9278278Z Generating XML reports... 2022-08-17T13:02:39.9311968Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130236.xml 2022-08-17T13:02:41.2395379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:41.2395800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:41.2398934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:41.2399404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:41.3955939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7frd406 2022-08-17T13:02:41.3959894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7frd406/_remote_module_non_scriptable.py 2022-08-17T13:02:41.6694468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:41.6706190Z 2022-08-17T13:02:41.6706557Z Running tests... 2022-08-17T13:02:41.6707234Z ---------------------------------------------------------------------- 2022-08-17T13:02:41.9381481Z test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12829 2022-08-17T13:02:41.9411900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12830 2022-08-17T13:02:41.9441989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12831 2022-08-17T13:02:41.9475150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12832 2022-08-17T13:02:42.9524656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:42.9525353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:42.9526899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:42.9527523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:42.9528330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:42.9528972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:42.9529791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:42.9530449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:42.9844909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:42.9845565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:42.9848863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:42.9849544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:42.9946640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:42.9947588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:42.9950556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:42.9951189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:43.1106714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ybjq5u5 2022-08-17T13:02:43.1108012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ybjq5u5/_remote_module_non_scriptable.py 2022-08-17T13:02:43.1109085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g85c17m 2022-08-17T13:02:43.1111137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g85c17m/_remote_module_non_scriptable.py 2022-08-17T13:02:43.1430424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xmr_mii 2022-08-17T13:02:43.1431237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xmr_mii/_remote_module_non_scriptable.py 2022-08-17T13:02:43.1540707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddj5dwd1 2022-08-17T13:02:43.1542241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddj5dwd1/_remote_module_non_scriptable.py 2022-08-17T13:02:43.3668910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:43.3690286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:43.4024310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:43.4152282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:43.6266489Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6267100Z ValueError('Expected error') 2022-08-17T13:02:43.6267524Z Traceback (most recent call last): 2022-08-17T13:02:43.6268345Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6268943Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6269407Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6269732Z raise ValueError(expected_err) 2022-08-17T13:02:43.6269948Z ValueError: Expected error 2022-08-17T13:02:43.6270076Z 2022-08-17T13:02:43.6283087Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6283691Z ValueError('Expected error') 2022-08-17T13:02:43.6284110Z Traceback (most recent call last): 2022-08-17T13:02:43.6285138Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6285721Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6286702Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6287244Z raise ValueError(expected_err) 2022-08-17T13:02:43.6287478Z ValueError: Expected error 2022-08-17T13:02:43.6291636Z 2022-08-17T13:02:43.6305635Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6306201Z ValueError('Expected error') 2022-08-17T13:02:43.6306626Z Traceback (most recent call last): 2022-08-17T13:02:43.6307344Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6307958Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6308779Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6309339Z raise ValueError(expected_err) 2022-08-17T13:02:43.6309724Z ValueError: Expected error 2022-08-17T13:02:43.6309950Z 2022-08-17T13:02:43.6310120Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6310576Z ValueError('Expected error') 2022-08-17T13:02:43.6311195Z Traceback (most recent call last): 2022-08-17T13:02:43.6311922Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6312526Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6313517Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6314225Z raise ValueError(expected_err) 2022-08-17T13:02:43.6314695Z ValueError: Expected error 2022-08-17T13:02:43.6314979Z 2022-08-17T13:02:43.6355237Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6355795Z ValueError('Expected error') 2022-08-17T13:02:43.6356194Z Traceback (most recent call last): 2022-08-17T13:02:43.6356636Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6356982Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6357452Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6357767Z raise ValueError(expected_err) 2022-08-17T13:02:43.6357991Z ValueError: Expected error 2022-08-17T13:02:43.6358192Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6358445Z ValueError('Expected error') 2022-08-17T13:02:43.6358665Z Traceback (most recent call last): 2022-08-17T13:02:43.6359102Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6359461Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6359927Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6360231Z raise ValueError(expected_err) 2022-08-17T13:02:43.6360443Z ValueError: Expected error 2022-08-17T13:02:43.6368805Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6369501Z ValueError('Expected error') 2022-08-17T13:02:43.6369987Z Traceback (most recent call last): 2022-08-17T13:02:43.6372326Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6372953Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6373809Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6374375Z raise ValueError(expected_err) 2022-08-17T13:02:43.6374760Z ValueError: Expected error 2022-08-17T13:02:43.6375323Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6375743Z ValueError('Expected error') 2022-08-17T13:02:43.6376121Z Traceback (most recent call last): 2022-08-17T13:02:43.6376813Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6377400Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6378176Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6378719Z raise ValueError(expected_err) 2022-08-17T13:02:43.6379102Z ValueError: Expected error 2022-08-17T13:02:43.6379334Z 2022-08-17T13:02:43.6379481Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6379921Z ValueError('Expected error') 2022-08-17T13:02:43.6380295Z Traceback (most recent call last): 2022-08-17T13:02:43.6380953Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6381547Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6382346Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6382896Z raise ValueError(expected_err) 2022-08-17T13:02:43.6383368Z ValueError: Expected error 2022-08-17T13:02:43.6383595Z 2022-08-17T13:02:43.6383603Z 2022-08-17T13:02:43.6383611Z 2022-08-17T13:02:43.6383646Z 2022-08-17T13:02:43.6383807Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:43.6384334Z ValueError('Expected error') 2022-08-17T13:02:43.6384724Z Traceback (most recent call last): 2022-08-17T13:02:43.6385396Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6385986Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6386776Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6387298Z raise ValueError(expected_err) 2022-08-17T13:02:43.6387663Z ValueError: Expected error 2022-08-17T13:02:43.6387846Z 2022-08-17T13:02:43.6421988Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6422701Z ValueError('Expected error') 2022-08-17T13:02:43.6423216Z Traceback (most recent call last): 2022-08-17T13:02:43.6425032Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6425811Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6426687Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6427277Z raise ValueError(expected_err) 2022-08-17T13:02:43.6428012Z ValueError: Expected error 2022-08-17T13:02:43.6428358Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6428727Z ValueError('Expected error') 2022-08-17T13:02:43.6429083Z Traceback (most recent call last): 2022-08-17T13:02:43.6429831Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6430444Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6431209Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6431527Z raise ValueError(expected_err) 2022-08-17T13:02:43.6431727Z ValueError: Expected error 2022-08-17T13:02:43.6436625Z 2022-08-17T13:02:43.6436797Z 2022-08-17T13:02:43.6439967Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6440677Z ValueError('Expected error') 2022-08-17T13:02:43.6441076Z Traceback (most recent call last): 2022-08-17T13:02:43.6441758Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6442295Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6443278Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6443793Z raise ValueError(expected_err) 2022-08-17T13:02:43.6444134Z ValueError: Expected error 2022-08-17T13:02:43.6444355Z 2022-08-17T13:02:43.6444561Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6445114Z ValueError('Expected error') 2022-08-17T13:02:43.6445563Z Traceback (most recent call last): 2022-08-17T13:02:43.6446691Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6447455Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6448460Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6449165Z raise ValueError(expected_err) 2022-08-17T13:02:43.6449647Z ValueError: Expected error 2022-08-17T13:02:43.6449941Z 2022-08-17T13:02:43.6450166Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6450701Z ValueError('Expected error') 2022-08-17T13:02:43.6451183Z Traceback (most recent call last): 2022-08-17T13:02:43.6452079Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6452979Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6454012Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6469513Z raise ValueError(expected_err) 2022-08-17T13:02:43.6470513Z ValueError: Expected error 2022-08-17T13:02:43.6470809Z 2022-08-17T13:02:43.6471026Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6472307Z ValueError('Expected error') 2022-08-17T13:02:43.6472792Z Traceback (most recent call last): 2022-08-17T13:02:43.6473699Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6474493Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6475532Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6476218Z raise ValueError(expected_err) 2022-08-17T13:02:43.6476686Z ValueError: Expected error 2022-08-17T13:02:43.6476964Z 2022-08-17T13:02:43.6477169Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6477719Z ValueError('Expected error') 2022-08-17T13:02:43.6478168Z Traceback (most recent call last): 2022-08-17T13:02:43.6479051Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6479803Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6480812Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6481517Z raise ValueError(expected_err) 2022-08-17T13:02:43.6481995Z ValueError: Expected error 2022-08-17T13:02:43.6482268Z 2022-08-17T13:02:43.6482450Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6482995Z ValueError('Expected error') 2022-08-17T13:02:43.6483462Z Traceback (most recent call last): 2022-08-17T13:02:43.6484364Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6485096Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6486294Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6487003Z raise ValueError(expected_err) 2022-08-17T13:02:43.6487453Z ValueError: Expected error 2022-08-17T13:02:43.6487727Z 2022-08-17T13:02:43.6487931Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6488486Z ValueError('Expected error') 2022-08-17T13:02:43.6489147Z Traceback (most recent call last): 2022-08-17T13:02:43.6490036Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6490792Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6491827Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6492509Z raise ValueError(expected_err) 2022-08-17T13:02:43.6492986Z ValueError: Expected error 2022-08-17T13:02:43.6493259Z 2022-08-17T13:02:43.6493462Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:43.6493987Z ValueError('Expected error') 2022-08-17T13:02:43.6494455Z Traceback (most recent call last): 2022-08-17T13:02:43.6495343Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6496080Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6497106Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6497800Z raise ValueError(expected_err) 2022-08-17T13:02:43.6498262Z ValueError: Expected error 2022-08-17T13:02:43.6498515Z 2022-08-17T13:02:43.6571637Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6572550Z ValueError('Expected error') 2022-08-17T13:02:43.6573049Z Traceback (most recent call last): 2022-08-17T13:02:43.6573948Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6574718Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6575758Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6576468Z raise ValueError(expected_err) 2022-08-17T13:02:43.6576918Z ValueError: Expected error 2022-08-17T13:02:43.6577868Z 2022-08-17T13:02:43.6587463Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6588048Z ValueError('Expected error') 2022-08-17T13:02:43.6588522Z Traceback (most recent call last): 2022-08-17T13:02:43.6589416Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6590185Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6591219Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6591904Z raise ValueError(expected_err) 2022-08-17T13:02:43.6592377Z ValueError: Expected error 2022-08-17T13:02:43.6592842Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6593734Z ValueError('Expected error') 2022-08-17T13:02:43.6594151Z Traceback (most recent call last): 2022-08-17T13:02:43.6594848Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6595401Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6596221Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6596747Z raise ValueError(expected_err) 2022-08-17T13:02:43.6596974Z ValueError: Expected error 2022-08-17T13:02:43.6597130Z 2022-08-17T13:02:43.6597213Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6597540Z ValueError('Expected error') 2022-08-17T13:02:43.6597884Z Traceback (most recent call last): 2022-08-17T13:02:43.6598518Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6599133Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6599948Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6600501Z raise ValueError(expected_err) 2022-08-17T13:02:43.6601048Z ValueError: Expected error 2022-08-17T13:02:43.6601276Z 2022-08-17T13:02:43.6601459Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6601927Z ValueError('Expected error') 2022-08-17T13:02:43.6602278Z Traceback (most recent call last): 2022-08-17T13:02:43.6602952Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6603487Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6604240Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6604684Z raise ValueError(expected_err) 2022-08-17T13:02:43.6605014Z ValueError: Expected error 2022-08-17T13:02:43.6605201Z 2022-08-17T13:02:43.6605373Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6605802Z ValueError('Expected error') 2022-08-17T13:02:43.6606386Z Traceback (most recent call last): 2022-08-17T13:02:43.6607064Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6607621Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6608537Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6609077Z raise ValueError(expected_err) 2022-08-17T13:02:43.6609415Z ValueError: Expected error 2022-08-17T13:02:43.6609617Z 2022-08-17T13:02:43.6609737Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6610173Z ValueError('Expected error') 2022-08-17T13:02:43.6610503Z Traceback (most recent call last): 2022-08-17T13:02:43.6611159Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6611697Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6612422Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6612945Z raise ValueError(expected_err) 2022-08-17T13:02:43.6613270Z ValueError: Expected error 2022-08-17T13:02:43.6613616Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6614028Z ValueError('Expected error') 2022-08-17T13:02:43.6614379Z Traceback (most recent call last): 2022-08-17T13:02:43.6615019Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6615455Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6615928Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6616232Z raise ValueError(expected_err) 2022-08-17T13:02:43.6616449Z ValueError: Expected error 2022-08-17T13:02:43.6616660Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6616959Z ValueError('Expected error') 2022-08-17T13:02:43.6617310Z Traceback (most recent call last): 2022-08-17T13:02:43.6617977Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6618517Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6619322Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6619872Z raise ValueError(expected_err) 2022-08-17T13:02:43.6620238Z ValueError: Expected error 2022-08-17T13:02:43.6620434Z 2022-08-17T13:02:43.6620576Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6620960Z ValueError('Expected error') 2022-08-17T13:02:43.6621311Z Traceback (most recent call last): 2022-08-17T13:02:43.6621959Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6622564Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6623441Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6624023Z raise ValueError(expected_err) 2022-08-17T13:02:43.6624335Z ValueError: Expected error 2022-08-17T13:02:43.6624539Z 2022-08-17T13:02:43.6624545Z 2022-08-17T13:02:43.6624690Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6625136Z ValueError('Expected error') 2022-08-17T13:02:43.6625500Z Traceback (most recent call last): 2022-08-17T13:02:43.6626133Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6626491Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6626960Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6627268Z raise ValueError(expected_err) 2022-08-17T13:02:43.6627489Z ValueError: Expected error 2022-08-17T13:02:43.6627620Z 2022-08-17T13:02:43.6627713Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6627954Z ValueError('Expected error') 2022-08-17T13:02:43.6628173Z Traceback (most recent call last): 2022-08-17T13:02:43.6628569Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6628997Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6629452Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6629767Z raise ValueError(expected_err) 2022-08-17T13:02:43.6629985Z ValueError: Expected error 2022-08-17T13:02:43.6630115Z 2022-08-17T13:02:43.6630194Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6630443Z ValueError('Expected error') 2022-08-17T13:02:43.6630663Z Traceback (most recent call last): 2022-08-17T13:02:43.6631047Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6631394Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6631854Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6632169Z raise ValueError(expected_err) 2022-08-17T13:02:43.6632372Z ValueError: Expected error 2022-08-17T13:02:43.6632583Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6632833Z ValueError('Expected error') 2022-08-17T13:02:43.6633037Z Traceback (most recent call last): 2022-08-17T13:02:43.6633436Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6633781Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6634230Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6634548Z raise ValueError(expected_err) 2022-08-17T13:02:43.6634764Z ValueError: Expected error 2022-08-17T13:02:43.6634890Z 2022-08-17T13:02:43.6634982Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6635219Z ValueError('Expected error') 2022-08-17T13:02:43.6635436Z Traceback (most recent call last): 2022-08-17T13:02:43.6635837Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6636170Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6636624Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6636937Z raise ValueError(expected_err) 2022-08-17T13:02:43.6637152Z ValueError: Expected error 2022-08-17T13:02:43.6637264Z 2022-08-17T13:02:43.6637269Z 2022-08-17T13:02:43.6637274Z 2022-08-17T13:02:43.6637365Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:43.6637611Z ValueError('Expected error') 2022-08-17T13:02:43.6637875Z Traceback (most recent call last): 2022-08-17T13:02:43.6638260Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6638607Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6639071Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6639387Z raise ValueError(expected_err) 2022-08-17T13:02:43.6639590Z ValueError: Expected error 2022-08-17T13:02:43.6639718Z 2022-08-17T13:02:43.6639814Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6640064Z ValueError('Expected error') 2022-08-17T13:02:43.6640268Z Traceback (most recent call last): 2022-08-17T13:02:43.6640668Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6641011Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6641457Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6641767Z raise ValueError(expected_err) 2022-08-17T13:02:43.6641982Z ValueError: Expected error 2022-08-17T13:02:43.6642108Z 2022-08-17T13:02:43.6642230Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6642466Z ValueError('Expected error') 2022-08-17T13:02:43.6642683Z Traceback (most recent call last): 2022-08-17T13:02:43.6643080Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6643410Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6643867Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6644180Z raise ValueError(expected_err) 2022-08-17T13:02:43.6644380Z ValueError: Expected error 2022-08-17T13:02:43.6644507Z 2022-08-17T13:02:43.6644598Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6644844Z ValueError('Expected error') 2022-08-17T13:02:43.6645061Z Traceback (most recent call last): 2022-08-17T13:02:43.6645441Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6645786Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6646443Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6646762Z raise ValueError(expected_err) 2022-08-17T13:02:43.6647098Z ValueError: Expected error 2022-08-17T13:02:43.6647286Z 2022-08-17T13:02:43.6647311Z 2022-08-17T13:02:43.6647468Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:43.6647845Z ValueError('Expected error') 2022-08-17T13:02:43.6648164Z Traceback (most recent call last): 2022-08-17T13:02:43.6648741Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:43.6649277Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:43.6649949Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:43.6650278Z raise ValueError(expected_err) 2022-08-17T13:02:43.6650497Z ValueError: Expected error 2022-08-17T13:02:43.6650626Z 2022-08-17T13:02:44.0533682Z ok (2.382s) 2022-08-17T13:02:44.0533881Z 2022-08-17T13:02:44.0534202Z ---------------------------------------------------------------------- 2022-08-17T13:02:44.0534467Z Ran 1 test in 2.383s 2022-08-17T13:02:44.0534590Z 2022-08-17T13:02:44.0534655Z OK 2022-08-17T13:02:44.0534753Z 2022-08-17T13:02:44.0534834Z Generating XML reports... 2022-08-17T13:02:44.0567891Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130241.xml 2022-08-17T13:02:45.3487102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:45.3487802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:45.3490509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:45.3491133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:45.5001981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev9d4y36 2022-08-17T13:02:45.5005326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev9d4y36/_remote_module_non_scriptable.py 2022-08-17T13:02:45.7665980Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:45.7677038Z 2022-08-17T13:02:45.7677349Z Running tests... 2022-08-17T13:02:45.7678041Z ---------------------------------------------------------------------- 2022-08-17T13:02:46.0353241Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13052 2022-08-17T13:02:46.0383272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13053 2022-08-17T13:02:46.0413680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13054 2022-08-17T13:02:46.0445461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13055 2022-08-17T13:02:47.0180948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:47.0181648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:47.0183840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:47.0184608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:47.0705541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:47.0706246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:47.0708358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:47.0709002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:47.0811257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:47.0811931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:47.0815577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:47.0816282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:47.1048288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:47.1048986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:47.1052896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:47.1053587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:47.1775974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvjaq5o7 2022-08-17T13:02:47.1777367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvjaq5o7/_remote_module_non_scriptable.py 2022-08-17T13:02:47.2326119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84bx1kpg 2022-08-17T13:02:47.2327208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84bx1kpg/_remote_module_non_scriptable.py 2022-08-17T13:02:47.2414133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65779ccg 2022-08-17T13:02:47.2415162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65779ccg/_remote_module_non_scriptable.py 2022-08-17T13:02:47.2662674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_1_whqn 2022-08-17T13:02:47.2663433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_1_whqn/_remote_module_non_scriptable.py 2022-08-17T13:02:47.4421529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:47.4969713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:47.5058067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:47.5337337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:47.7499508Z On WorkerInfo(id=1, name=worker1): 2022-08-17T13:02:47.7500654Z ValueError('Expected error') 2022-08-17T13:02:47.7501703Z Traceback (most recent call last): 2022-08-17T13:02:47.7502829Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:47.7503633Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:47.7505241Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:47.7505979Z raise ValueError(expected_err) 2022-08-17T13:02:47.7506450Z ValueError: Expected error 2022-08-17T13:02:47.7506746Z 2022-08-17T13:02:47.7712500Z On WorkerInfo(id=0, name=worker0): 2022-08-17T13:02:47.7716507Z ValueError('Expected error') 2022-08-17T13:02:47.7716890Z Traceback (most recent call last): 2022-08-17T13:02:47.7717632Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:47.7718226Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:47.7718886Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:47.7719368Z raise ValueError(expected_err) 2022-08-17T13:02:47.7719643Z ValueError: Expected error 2022-08-17T13:02:47.7719867Z 2022-08-17T13:02:47.7819735Z On WorkerInfo(id=3, name=worker3): 2022-08-17T13:02:47.7820552Z ValueError('Expected error') 2022-08-17T13:02:47.7821018Z Traceback (most recent call last): 2022-08-17T13:02:47.7821973Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:47.7822755Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:47.7823791Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:47.7824582Z raise ValueError(expected_err) 2022-08-17T13:02:47.7825068Z ValueError: Expected error 2022-08-17T13:02:47.7825360Z 2022-08-17T13:02:47.7932378Z On WorkerInfo(id=2, name=worker2): 2022-08-17T13:02:47.7933275Z ValueError('Expected error') 2022-08-17T13:02:47.7933940Z Traceback (most recent call last): 2022-08-17T13:02:47.7935069Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-08-17T13:02:47.7936021Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-08-17T13:02:47.7937426Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 398, in raise_func 2022-08-17T13:02:47.7938051Z raise ValueError(expected_err) 2022-08-17T13:02:47.7938716Z ValueError: Expected error 2022-08-17T13:02:47.7938987Z 2022-08-17T13:02:48.1503828Z ok (2.382s) 2022-08-17T13:02:48.1504160Z 2022-08-17T13:02:48.1504593Z ---------------------------------------------------------------------- 2022-08-17T13:02:48.1504876Z Ran 1 test in 2.383s 2022-08-17T13:02:48.1504984Z 2022-08-17T13:02:48.1505051Z OK 2022-08-17T13:02:48.1505147Z 2022-08-17T13:02:48.1505482Z Generating XML reports... 2022-08-17T13:02:48.1538828Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130245.xml 2022-08-17T13:02:49.4527828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:49.4528246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:49.4530944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:49.4531345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:49.6036546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9p2nugs 2022-08-17T13:02:49.6040245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9p2nugs/_remote_module_non_scriptable.py 2022-08-17T13:02:49.8661001Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:49.8672227Z 2022-08-17T13:02:49.8672453Z Running tests... 2022-08-17T13:02:49.8673183Z ---------------------------------------------------------------------- 2022-08-17T13:02:50.1343674Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13275 2022-08-17T13:02:50.1373151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13276 2022-08-17T13:02:50.1401884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13277 2022-08-17T13:02:50.1433527Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13278 2022-08-17T13:02:51.1970488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:51.1971182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:51.1974635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:51.1975365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:51.2064721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:51.2065351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:51.2068694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:51.2069345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:51.2608356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:51.2608886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:51.2611962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:51.2612512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:51.2757202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:51.2757955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:51.2759763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:51.2760511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:51.3590576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89ycazt1 2022-08-17T13:02:51.3591302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89ycazt1/_remote_module_non_scriptable.py 2022-08-17T13:02:51.3661957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkzjcmup 2022-08-17T13:02:51.3663066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkzjcmup/_remote_module_non_scriptable.py 2022-08-17T13:02:51.4229466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69b01qqo 2022-08-17T13:02:51.4230205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69b01qqo/_remote_module_non_scriptable.py 2022-08-17T13:02:51.4356396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizhjmihw 2022-08-17T13:02:51.4357542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizhjmihw/_remote_module_non_scriptable.py 2022-08-17T13:02:51.6240338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:51.6279250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:51.6885384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:51.6997733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:51.7108650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:02:51.7300191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:02:51.7301143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:02:51.7301725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:02:51.7302383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:51.7302934Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:51.7303483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:51.7312270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:52.1673163Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.1674141Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.1674758Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.1693109Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.1694193Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.1695241Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:52.5498322Z ok (2.682s) 2022-08-17T13:02:52.5498577Z 2022-08-17T13:02:52.5499001Z ---------------------------------------------------------------------- 2022-08-17T13:02:52.5499464Z Ran 1 test in 2.683s 2022-08-17T13:02:52.5499688Z 2022-08-17T13:02:52.5499809Z OK 2022-08-17T13:02:52.5499925Z 2022-08-17T13:02:52.5500026Z Generating XML reports... 2022-08-17T13:02:52.5533484Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130249.xml 2022-08-17T13:02:53.8755524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:53.8756190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:53.8758161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:53.8758538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:54.0267868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlumf79r 2022-08-17T13:02:54.0271263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlumf79r/_remote_module_non_scriptable.py 2022-08-17T13:02:54.2915581Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:02:54.2926046Z 2022-08-17T13:02:54.2926201Z Running tests... 2022-08-17T13:02:54.2926829Z ---------------------------------------------------------------------- 2022-08-17T13:02:54.5568670Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13510 2022-08-17T13:02:54.5598803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13511 2022-08-17T13:02:54.5627309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13512 2022-08-17T13:02:54.5658727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13513 2022-08-17T13:02:55.5401158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:55.5401808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:55.5402610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:55.5403231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:55.5404285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:55.5406948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:55.5407892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:55.5408519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:55.5800262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:55.5800953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:55.5803977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:55.5804725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:55.5807144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:02:55.5807836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:02:55.5811629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:02:55.5812333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:02:55.6989456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mo845lr 2022-08-17T13:02:55.6990302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq8meczl 2022-08-17T13:02:55.6991415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mo845lr/_remote_module_non_scriptable.py 2022-08-17T13:02:55.6992197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq8meczl/_remote_module_non_scriptable.py 2022-08-17T13:02:55.7382570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsw5jps7 2022-08-17T13:02:55.7383449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsw5jps7/_remote_module_non_scriptable.py 2022-08-17T13:02:55.7399183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d96gywe 2022-08-17T13:02:55.7401743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d96gywe/_remote_module_non_scriptable.py 2022-08-17T13:02:55.9599436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:02:55.9606221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:02:56.0001353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:02:56.0037333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:02:56.0270879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:02:56.0370169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:02:56.0473104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:02:56.0473732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:02:56.0474663Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:56.0475323Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:56.0475888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:56.0476489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:02:56.8157493Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:56.8161521Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:56.8254164Z [W tensorpipe_agent.cpp:940] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:56.8255129Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:56.8256333Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:56.8257231Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:56.8258318Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:02:57.3066384Z [W tensorpipe_agent.cpp:916] RPC agent for worker0 encountered error when sending outgoing request #4 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:57.3067432Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down 2022-08-17T13:02:57.3068146Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:57.3158951Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:57.3163977Z [W tensorpipe_agent.cpp:679] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:02:59.6762542Z ok (5.383s) 2022-08-17T13:02:59.6762812Z 2022-08-17T13:02:59.6763404Z ---------------------------------------------------------------------- 2022-08-17T13:02:59.6763833Z Ran 1 test in 5.384s 2022-08-17T13:02:59.6764020Z 2022-08-17T13:02:59.6764132Z OK 2022-08-17T13:02:59.6764271Z 2022-08-17T13:02:59.6764419Z Generating XML reports... 2022-08-17T13:02:59.6798612Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130254.xml 2022-08-17T13:03:01.0060025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:01.0060434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:01.0063101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:01.0063881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:01.1605697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgvn2oeu 2022-08-17T13:03:01.1609592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgvn2oeu/_remote_module_non_scriptable.py 2022-08-17T13:03:01.4345516Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:01.4356153Z 2022-08-17T13:03:01.4356286Z Running tests... 2022-08-17T13:03:01.4356683Z ---------------------------------------------------------------------- 2022-08-17T13:03:01.7020068Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13745 2022-08-17T13:03:01.7049182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13746 2022-08-17T13:03:01.7079077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13747 2022-08-17T13:03:01.7111755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13748 2022-08-17T13:03:02.7200616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:02.7201231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:02.7203444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:02.7204570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:02.7398198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:02.7398865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:02.7402031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:02.7402746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:02.7685555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:02.7686446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:02.7689477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:02.7690131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:02.7787148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:02.7788186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:02.7791452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:02.7792034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:02.8815878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_n1nv0a 2022-08-17T13:03:02.8816925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_n1nv0a/_remote_module_non_scriptable.py 2022-08-17T13:03:02.9013267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4opp2eo 2022-08-17T13:03:02.9014672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4opp2eo/_remote_module_non_scriptable.py 2022-08-17T13:03:02.9294045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7b9w14s 2022-08-17T13:03:02.9295154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7b9w14s/_remote_module_non_scriptable.py 2022-08-17T13:03:02.9404049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllqki1uo 2022-08-17T13:03:02.9405378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllqki1uo/_remote_module_non_scriptable.py 2022-08-17T13:03:03.1457811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:03.1617316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:03.1916043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:03.2036489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:03.2227549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:03:03.2333556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:03:03.2333996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:03:03.2334392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:03:03.2335049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:03:03.2335609Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:03:03.2336148Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:03:03.2431402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:03:03.6551899Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:03.6552880Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:03.6554004Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:03.6604130Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:03.6604720Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:03.6605280Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:03:04.0173853Z ok (2.581s) 2022-08-17T13:03:04.0174154Z 2022-08-17T13:03:04.0174650Z ---------------------------------------------------------------------- 2022-08-17T13:03:04.0174921Z Ran 1 test in 2.582s 2022-08-17T13:03:04.0175046Z 2022-08-17T13:03:04.0175110Z OK 2022-08-17T13:03:04.0175205Z 2022-08-17T13:03:04.0175299Z Generating XML reports... 2022-08-17T13:03:04.0208361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130301.xml 2022-08-17T13:03:05.2987221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:05.2987630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:05.2990168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:05.2990569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:05.4470578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wyj9nay 2022-08-17T13:03:05.4473857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wyj9nay/_remote_module_non_scriptable.py 2022-08-17T13:03:05.7216020Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:05.7227098Z 2022-08-17T13:03:05.7227237Z Running tests... 2022-08-17T13:03:05.7227678Z ---------------------------------------------------------------------- 2022-08-17T13:03:05.9864584Z test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13980 2022-08-17T13:03:05.9893488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13981 2022-08-17T13:03:05.9921228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13982 2022-08-17T13:03:05.9954075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13983 2022-08-17T13:03:07.0607947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:07.0608604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:07.0611176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:07.0611887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:07.0890195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:07.0890864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:07.0893868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:07.0894525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:07.1163307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:07.1163970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:07.1166298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:07.1166935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:07.1245346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:07.1246203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:07.1250036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:07.1250580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:07.2214720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sn2c45_ 2022-08-17T13:03:07.2215559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sn2c45_/_remote_module_non_scriptable.py 2022-08-17T13:03:07.2471315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqfab0nl 2022-08-17T13:03:07.2472177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqfab0nl/_remote_module_non_scriptable.py 2022-08-17T13:03:07.2759389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiew0un3a 2022-08-17T13:03:07.2760299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiew0un3a/_remote_module_non_scriptable.py 2022-08-17T13:03:07.2844353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m74qun6 2022-08-17T13:03:07.2845079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m74qun6/_remote_module_non_scriptable.py 2022-08-17T13:03:07.4849696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:07.5093949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:07.5387792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:07.5465557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:08.2016043Z ok (2.479s) 2022-08-17T13:03:08.2016350Z 2022-08-17T13:03:08.2016855Z ---------------------------------------------------------------------- 2022-08-17T13:03:08.2017355Z Ran 1 test in 2.479s 2022-08-17T13:03:08.2017521Z 2022-08-17T13:03:08.2017585Z OK 2022-08-17T13:03:08.2017668Z 2022-08-17T13:03:08.2017764Z Generating XML reports... 2022-08-17T13:03:08.2051400Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130305.xml 2022-08-17T13:03:09.5162737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:09.5163137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:09.5165708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:09.5166255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:09.6716809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqt_ljuol 2022-08-17T13:03:09.6719999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqt_ljuol/_remote_module_non_scriptable.py 2022-08-17T13:03:09.9337122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:09.9347912Z 2022-08-17T13:03:09.9348070Z Running tests... 2022-08-17T13:03:09.9349262Z ---------------------------------------------------------------------- 2022-08-17T13:03:10.1932207Z test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14203 2022-08-17T13:03:10.1960471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14204 2022-08-17T13:03:10.1989395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14205 2022-08-17T13:03:10.2019997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14206 2022-08-17T13:03:11.2704402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:11.2705052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:11.2708114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:11.2708808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:11.3103486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:11.3104239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:11.3105542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:11.3106158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:11.3108195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:11.3109116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:11.3110450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:11.3111092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:11.3205398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:11.3206307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:11.3209907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:11.3210829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:11.4283946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkghw5wc2 2022-08-17T13:03:11.4284771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkghw5wc2/_remote_module_non_scriptable.py 2022-08-17T13:03:11.4690054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9o45fjk 2022-08-17T13:03:11.4690896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9o45fjk/_remote_module_non_scriptable.py 2022-08-17T13:03:11.4708429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8jmu3dv 2022-08-17T13:03:11.4710357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8jmu3dv/_remote_module_non_scriptable.py 2022-08-17T13:03:11.4806252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n39uegy 2022-08-17T13:03:11.4808653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n39uegy/_remote_module_non_scriptable.py 2022-08-17T13:03:11.6903736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:11.7277783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:11.7321168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:11.7427249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:12.4080556Z ok (2.473s) 2022-08-17T13:03:12.4080835Z 2022-08-17T13:03:12.4081260Z ---------------------------------------------------------------------- 2022-08-17T13:03:12.4081544Z Ran 1 test in 2.473s 2022-08-17T13:03:12.4081668Z 2022-08-17T13:03:12.4081738Z OK 2022-08-17T13:03:12.4081820Z 2022-08-17T13:03:12.4081918Z Generating XML reports... 2022-08-17T13:03:12.4115494Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130309.xml 2022-08-17T13:03:13.7223688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:13.7224142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:13.7226813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:13.7227200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:13.8739717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ltlvphs 2022-08-17T13:03:13.8742816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ltlvphs/_remote_module_non_scriptable.py 2022-08-17T13:03:14.1343362Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:14.1354106Z 2022-08-17T13:03:14.1354248Z Running tests... 2022-08-17T13:03:14.1354706Z ---------------------------------------------------------------------- 2022-08-17T13:03:14.3986740Z test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14426 2022-08-17T13:03:14.4014509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14427 2022-08-17T13:03:14.4043425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14428 2022-08-17T13:03:14.4074428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14429 2022-08-17T13:03:15.4505292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:15.4505912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:15.4509004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:15.4509959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:15.4917165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:15.4917836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:15.4921414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:15.4922072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:15.5226259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:15.5227029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:15.5229620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:15.5230364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:15.5304627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:15.5305236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:15.5308512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:15.5309200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:15.6079469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkrms7jc 2022-08-17T13:03:15.6080261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkrms7jc/_remote_module_non_scriptable.py 2022-08-17T13:03:15.6506034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdsxn3ay 2022-08-17T13:03:15.6507640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdsxn3ay/_remote_module_non_scriptable.py 2022-08-17T13:03:15.6801329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mfsb2a3 2022-08-17T13:03:15.6802698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mfsb2a3/_remote_module_non_scriptable.py 2022-08-17T13:03:15.6889435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mhe84v1 2022-08-17T13:03:15.6891065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mhe84v1/_remote_module_non_scriptable.py 2022-08-17T13:03:15.8698564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:15.9312003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:15.9472045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:15.9632781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:16.3130816Z ok (2.177s) 2022-08-17T13:03:16.3131102Z 2022-08-17T13:03:16.3131621Z ---------------------------------------------------------------------- 2022-08-17T13:03:16.3131896Z Ran 1 test in 2.178s 2022-08-17T13:03:16.3132019Z 2022-08-17T13:03:16.3132084Z OK 2022-08-17T13:03:16.3132185Z 2022-08-17T13:03:16.3132282Z Generating XML reports... 2022-08-17T13:03:16.3167227Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130314.xml 2022-08-17T13:03:17.6086882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:17.6087293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:17.6089606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:17.6090020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:17.7598158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdh2k3ha 2022-08-17T13:03:17.7602146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdh2k3ha/_remote_module_non_scriptable.py 2022-08-17T13:03:18.0279570Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:18.0290853Z 2022-08-17T13:03:18.0291158Z Running tests... 2022-08-17T13:03:18.0291816Z ---------------------------------------------------------------------- 2022-08-17T13:03:18.2949913Z test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14523 2022-08-17T13:03:18.2979741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14524 2022-08-17T13:03:18.3009774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14525 2022-08-17T13:03:18.3042068Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14526 2022-08-17T13:03:19.4230313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:19.4230977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:19.4231763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:19.4232349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:19.4233133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:19.4233776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:19.4234613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:19.4235255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:19.4444270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:19.4444928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:19.4447850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:19.4448515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:19.4512107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:19.4516725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:19.4517572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:19.5818107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:19.5818740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9xlbe8y 2022-08-17T13:03:19.5819499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9xlbe8y/_remote_module_non_scriptable.py 2022-08-17T13:03:19.5830989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw0_qo4s 2022-08-17T13:03:19.5833487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw0_qo4s/_remote_module_non_scriptable.py 2022-08-17T13:03:19.6039471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmxm39gj 2022-08-17T13:03:19.6040296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmxm39gj/_remote_module_non_scriptable.py 2022-08-17T13:03:19.6115098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx019l2k 2022-08-17T13:03:19.6116243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx019l2k/_remote_module_non_scriptable.py 2022-08-17T13:03:19.8394980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:19.8431822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:19.8641604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:19.8733719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:20.2097584Z ok (2.180s) 2022-08-17T13:03:20.2097850Z 2022-08-17T13:03:20.2098376Z ---------------------------------------------------------------------- 2022-08-17T13:03:20.2098826Z Ran 1 test in 2.181s 2022-08-17T13:03:20.2098949Z 2022-08-17T13:03:20.2099001Z OK 2022-08-17T13:03:20.2099094Z 2022-08-17T13:03:20.2099187Z Generating XML reports... 2022-08-17T13:03:20.2132065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130318.xml 2022-08-17T13:03:21.5089940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:21.5090362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:21.5092926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:21.5093307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:21.6602788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2bvhvek 2022-08-17T13:03:21.6606072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2bvhvek/_remote_module_non_scriptable.py 2022-08-17T13:03:21.9232073Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:21.9242593Z 2022-08-17T13:03:21.9242690Z Running tests... 2022-08-17T13:03:21.9243429Z ---------------------------------------------------------------------- 2022-08-17T13:03:22.1875455Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14578 2022-08-17T13:03:22.1904282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14579 2022-08-17T13:03:22.1936425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14580 2022-08-17T13:03:22.1966987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14581 2022-08-17T13:03:23.1704368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:23.1705022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:23.1708659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:23.1710011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:23.1796942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:23.1797569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:23.1800608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:23.1801215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:23.1843663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:23.1844350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:23.1848810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:23.1849591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:23.2582604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:23.2583351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:23.2586523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:23.2587271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:23.3324186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l3atn8t 2022-08-17T13:03:23.3325739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l3atn8t/_remote_module_non_scriptable.py 2022-08-17T13:03:23.3416491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d5s9_b7 2022-08-17T13:03:23.3417383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d5s9_b7/_remote_module_non_scriptable.py 2022-08-17T13:03:23.3521649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdjpl8cp 2022-08-17T13:03:23.3522463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdjpl8cp/_remote_module_non_scriptable.py 2022-08-17T13:03:23.4193669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yhgb2iz 2022-08-17T13:03:23.4196784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yhgb2iz/_remote_module_non_scriptable.py 2022-08-17T13:03:23.5955446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:23.6063382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:23.6159315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:23.6841289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:24.3024831Z ok (2.378s) 2022-08-17T13:03:24.3025035Z 2022-08-17T13:03:24.3025558Z ---------------------------------------------------------------------- 2022-08-17T13:03:24.3025829Z Ran 1 test in 2.378s 2022-08-17T13:03:24.3025951Z 2022-08-17T13:03:24.3026022Z OK 2022-08-17T13:03:24.3026104Z 2022-08-17T13:03:24.3026218Z Generating XML reports... 2022-08-17T13:03:24.3059678Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130321.xml 2022-08-17T13:03:25.5899083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:25.5899742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:25.5902616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:25.5903222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:25.7398285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf5ki7e4 2022-08-17T13:03:25.7401190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf5ki7e4/_remote_module_non_scriptable.py 2022-08-17T13:03:26.0048037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:26.0058562Z 2022-08-17T13:03:26.0058874Z Running tests... 2022-08-17T13:03:26.0059530Z ---------------------------------------------------------------------- 2022-08-17T13:03:26.2674612Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14833 2022-08-17T13:03:26.2702558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14834 2022-08-17T13:03:26.2731138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14835 2022-08-17T13:03:26.2763870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14836 2022-08-17T13:03:27.2849312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:27.2849960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:27.2852856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:27.2853603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:27.3387543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:27.3388246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:27.3392439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:27.3393096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:27.3426530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:27.3427185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:27.3431159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:27.3431826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:27.3677599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:27.3678247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:27.3681694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:27.3682391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:27.4466277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuoosux57 2022-08-17T13:03:27.4467045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuoosux57/_remote_module_non_scriptable.py 2022-08-17T13:03:27.5031957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv4fpcqh 2022-08-17T13:03:27.5032762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv4fpcqh/_remote_module_non_scriptable.py 2022-08-17T13:03:27.5165567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnx0oqut 2022-08-17T13:03:27.5167223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnx0oqut/_remote_module_non_scriptable.py 2022-08-17T13:03:27.5311278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ctfuod7 2022-08-17T13:03:27.5312284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ctfuod7/_remote_module_non_scriptable.py 2022-08-17T13:03:27.7042192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:27.7641421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:27.7762470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:27.7918792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:28.6826606Z ok (2.676s) 2022-08-17T13:03:28.6826878Z 2022-08-17T13:03:28.6827406Z ---------------------------------------------------------------------- 2022-08-17T13:03:28.6827871Z Ran 1 test in 2.677s 2022-08-17T13:03:28.6828003Z 2022-08-17T13:03:28.6828055Z OK 2022-08-17T13:03:28.6828149Z 2022-08-17T13:03:28.6828243Z Generating XML reports... 2022-08-17T13:03:28.6865578Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130325.xml 2022-08-17T13:03:29.9828305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:29.9828698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:29.9831268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:29.9831638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:30.1317326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptty1eqz9 2022-08-17T13:03:30.1320647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptty1eqz9/_remote_module_non_scriptable.py 2022-08-17T13:03:30.3927814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:30.3939365Z 2022-08-17T13:03:30.3939652Z Running tests... 2022-08-17T13:03:30.3940293Z ---------------------------------------------------------------------- 2022-08-17T13:03:30.6541456Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15136 2022-08-17T13:03:30.6570466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15137 2022-08-17T13:03:30.6599763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15138 2022-08-17T13:03:30.6633107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15139 2022-08-17T13:03:31.7604931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:31.7605608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:31.7607595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:31.7608223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:31.7824883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:31.7825556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:31.7828752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:31.7829417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:31.7993667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:31.7994368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:31.7997753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:31.7998381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:31.8067908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:31.8068585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:31.8069429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:31.8070091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:31.9193931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn40zz_7 2022-08-17T13:03:31.9194788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn40zz_7/_remote_module_non_scriptable.py 2022-08-17T13:03:31.9432423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6jnw_cr 2022-08-17T13:03:31.9434077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6jnw_cr/_remote_module_non_scriptable.py 2022-08-17T13:03:31.9592687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5bg1zta 2022-08-17T13:03:31.9593689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5bg1zta/_remote_module_non_scriptable.py 2022-08-17T13:03:31.9645214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqautcul4 2022-08-17T13:03:31.9647002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqautcul4/_remote_module_non_scriptable.py 2022-08-17T13:03:32.1821376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:32.2066305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:32.2203778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:32.2277126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:32.9693332Z ok (2.575s) 2022-08-17T13:03:32.9693535Z 2022-08-17T13:03:32.9693862Z ---------------------------------------------------------------------- 2022-08-17T13:03:32.9694165Z Ran 1 test in 2.575s 2022-08-17T13:03:32.9694289Z 2022-08-17T13:03:32.9694353Z OK 2022-08-17T13:03:32.9694449Z 2022-08-17T13:03:32.9694530Z Generating XML reports... 2022-08-17T13:03:32.9727947Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130330.xml 2022-08-17T13:03:34.2656888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:34.2657356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:34.2659886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:34.2660286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:34.4144814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyu70p4y 2022-08-17T13:03:34.4148676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyu70p4y/_remote_module_non_scriptable.py 2022-08-17T13:03:34.6746614Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:34.6756942Z 2022-08-17T13:03:34.6757108Z Running tests... 2022-08-17T13:03:34.6757535Z ---------------------------------------------------------------------- 2022-08-17T13:03:34.9371331Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15391 2022-08-17T13:03:34.9398619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15392 2022-08-17T13:03:34.9427309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15393 2022-08-17T13:03:34.9459372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15394 2022-08-17T13:03:35.9295503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:35.9296170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:35.9298999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:35.9299695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:35.9370282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:35.9370936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:35.9374920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:35.9375593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:36.0058126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:36.0058898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:36.0062119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:36.0063181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:36.0264116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:36.0264897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:36.0267730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:36.0268400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:36.0880534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rcxtmua 2022-08-17T13:03:36.0881382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rcxtmua/_remote_module_non_scriptable.py 2022-08-17T13:03:36.0943414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvytqlrn6 2022-08-17T13:03:36.0945130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvytqlrn6/_remote_module_non_scriptable.py 2022-08-17T13:03:36.1639314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv69s4rb 2022-08-17T13:03:36.1640573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv69s4rb/_remote_module_non_scriptable.py 2022-08-17T13:03:36.1831245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ngwsj3t 2022-08-17T13:03:36.1832771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ngwsj3t/_remote_module_non_scriptable.py 2022-08-17T13:03:36.3497772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:36.3519912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:36.4267931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:36.4453776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:37.1517426Z ok (2.476s) 2022-08-17T13:03:37.1517682Z 2022-08-17T13:03:37.1518144Z ---------------------------------------------------------------------- 2022-08-17T13:03:37.1518639Z Ran 1 test in 2.476s 2022-08-17T13:03:37.1518830Z 2022-08-17T13:03:37.1518912Z OK 2022-08-17T13:03:37.1519009Z 2022-08-17T13:03:37.1519105Z Generating XML reports... 2022-08-17T13:03:37.1552066Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130334.xml 2022-08-17T13:03:38.4347383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:38.4348351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:38.4350930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:38.4351496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:38.5841600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vkg29hz 2022-08-17T13:03:38.5844932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vkg29hz/_remote_module_non_scriptable.py 2022-08-17T13:03:38.8495470Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:38.8506354Z 2022-08-17T13:03:38.8506530Z Running tests... 2022-08-17T13:03:38.8506876Z ---------------------------------------------------------------------- 2022-08-17T13:03:39.1193514Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15658 2022-08-17T13:03:39.1222396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15659 2022-08-17T13:03:39.1252898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15660 2022-08-17T13:03:39.1286272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15661 2022-08-17T13:03:40.1372315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:40.1372975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:40.1375805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:40.1376447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:40.1616366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:40.1617065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:40.1619916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:40.1620586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:40.1658005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:40.1658650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:40.1662425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:40.1663097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:40.2013081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:40.2013758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:40.2016708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:40.2017343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:40.2970684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7opl34j 2022-08-17T13:03:40.2971728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7opl34j/_remote_module_non_scriptable.py 2022-08-17T13:03:40.3229873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppaf5br8z 2022-08-17T13:03:40.3230703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppaf5br8z/_remote_module_non_scriptable.py 2022-08-17T13:03:40.3273623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2bdj4ei 2022-08-17T13:03:40.3275046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2bdj4ei/_remote_module_non_scriptable.py 2022-08-17T13:03:40.3607995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2e38w98 2022-08-17T13:03:40.3608697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2e38w98/_remote_module_non_scriptable.py 2022-08-17T13:03:40.5625193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:40.5902588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:40.5945223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:40.6284745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:41.5351157Z ok (2.684s) 2022-08-17T13:03:41.5351382Z 2022-08-17T13:03:41.5351943Z ---------------------------------------------------------------------- 2022-08-17T13:03:41.5352321Z Ran 1 test in 2.684s 2022-08-17T13:03:41.5352444Z 2022-08-17T13:03:41.5352526Z OK 2022-08-17T13:03:41.5352620Z 2022-08-17T13:03:41.5352716Z Generating XML reports... 2022-08-17T13:03:41.5388841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130338.xml 2022-08-17T13:03:42.8462133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:42.8462559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:42.8464706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:42.8465081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:43.0021586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo10_s7t9 2022-08-17T13:03:43.0024665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo10_s7t9/_remote_module_non_scriptable.py 2022-08-17T13:03:43.2704328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:43.2714854Z 2022-08-17T13:03:43.2715007Z Running tests... 2022-08-17T13:03:43.2716161Z ---------------------------------------------------------------------- 2022-08-17T13:03:43.5389218Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15937 2022-08-17T13:03:43.5418440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15938 2022-08-17T13:03:43.5447291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15939 2022-08-17T13:03:43.5479403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15940 2022-08-17T13:03:44.6334785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:44.6335412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:44.6338662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:44.6339382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:44.6518004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:44.6518690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:44.6521784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:44.6522464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:44.6822712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:44.6823409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:44.6826419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:44.6827126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:44.7168449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:44.7169070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:44.7171962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:44.7172580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:44.7962500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_2gwtig 2022-08-17T13:03:44.7963252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_2gwtig/_remote_module_non_scriptable.py 2022-08-17T13:03:44.8164704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddyfos6l 2022-08-17T13:03:44.8165629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddyfos6l/_remote_module_non_scriptable.py 2022-08-17T13:03:44.8469365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ijnt9ju 2022-08-17T13:03:44.8470376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ijnt9ju/_remote_module_non_scriptable.py 2022-08-17T13:03:44.8786725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv600tc1 2022-08-17T13:03:44.8787545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv600tc1/_remote_module_non_scriptable.py 2022-08-17T13:03:45.0669544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:45.0862391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:45.1166081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:45.1455062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:45.9594549Z ok (2.688s) 2022-08-17T13:03:45.9594774Z 2022-08-17T13:03:45.9595266Z ---------------------------------------------------------------------- 2022-08-17T13:03:45.9595750Z Ran 1 test in 2.688s 2022-08-17T13:03:45.9595878Z 2022-08-17T13:03:45.9595947Z OK 2022-08-17T13:03:45.9596041Z 2022-08-17T13:03:45.9596135Z Generating XML reports... 2022-08-17T13:03:45.9629394Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130343.xml 2022-08-17T13:03:47.2769069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:47.2769723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:47.2772162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:47.2772573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:47.4319018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsas00t6t 2022-08-17T13:03:47.4321754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsas00t6t/_remote_module_non_scriptable.py 2022-08-17T13:03:47.6960965Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:47.6971658Z 2022-08-17T13:03:47.6971768Z Running tests... 2022-08-17T13:03:47.6972545Z ---------------------------------------------------------------------- 2022-08-17T13:03:47.9639933Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16216 2022-08-17T13:03:47.9668912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16217 2022-08-17T13:03:47.9697503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16218 2022-08-17T13:03:47.9730520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16219 2022-08-17T13:03:48.9764937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:48.9765605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:48.9767869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:48.9768527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:49.0011167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:49.0011769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:49.0014188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:49.0014832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:49.0055786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:49.0056743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:49.0060086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:49.0060704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:49.0074309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:49.0074872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:49.0079396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:49.0080016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:49.1358491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7gp2meh 2022-08-17T13:03:49.1359348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7gp2meh/_remote_module_non_scriptable.py 2022-08-17T13:03:49.1608498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhd0jd1z 2022-08-17T13:03:49.1609747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhd0jd1z/_remote_module_non_scriptable.py 2022-08-17T13:03:49.1648535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghhspov7 2022-08-17T13:03:49.1650302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghhspov7/_remote_module_non_scriptable.py 2022-08-17T13:03:49.1679268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7ms9obd 2022-08-17T13:03:49.1681260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7ms9obd/_remote_module_non_scriptable.py 2022-08-17T13:03:49.4174400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:49.4212038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:49.4293674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:49.4310640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:50.2791652Z ok (2.582s) 2022-08-17T13:03:50.2791893Z 2022-08-17T13:03:50.2792444Z ---------------------------------------------------------------------- 2022-08-17T13:03:50.2792828Z Ran 1 test in 2.582s 2022-08-17T13:03:50.2792948Z 2022-08-17T13:03:50.2793012Z OK 2022-08-17T13:03:50.2793095Z 2022-08-17T13:03:50.2793190Z Generating XML reports... 2022-08-17T13:03:50.2827837Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130347.xml 2022-08-17T13:03:51.6091131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:51.6091574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:51.6093840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:51.6094222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:51.7632817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5qqdz2r 2022-08-17T13:03:51.7636124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5qqdz2r/_remote_module_non_scriptable.py 2022-08-17T13:03:52.0332235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:52.0342896Z 2022-08-17T13:03:52.0343010Z Running tests... 2022-08-17T13:03:52.0343473Z ---------------------------------------------------------------------- 2022-08-17T13:03:52.2968972Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16471 2022-08-17T13:03:52.2998466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16472 2022-08-17T13:03:52.3026120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16473 2022-08-17T13:03:52.3057281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16474 2022-08-17T13:03:53.3655031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:53.3655666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:53.3658690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:53.3659362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:53.3872436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:53.3873111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:53.3875529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:53.3876181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:53.4613541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:53.4615020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:53.4615899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:53.4616532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:53.5118178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:53.5118831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:53.5122657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:53.5123347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:53.5227261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvegtidim 2022-08-17T13:03:53.5228382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvegtidim/_remote_module_non_scriptable.py 2022-08-17T13:03:53.5476167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2edmrrps 2022-08-17T13:03:53.5477492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2edmrrps/_remote_module_non_scriptable.py 2022-08-17T13:03:53.6378375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2agtiny 2022-08-17T13:03:53.6379185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2agtiny/_remote_module_non_scriptable.py 2022-08-17T13:03:53.6744895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeqwqoku 2022-08-17T13:03:53.6745777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeqwqoku/_remote_module_non_scriptable.py 2022-08-17T13:03:53.7879210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:53.8134849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:53.9002955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:53.9365475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:54.8122142Z ok (2.778s) 2022-08-17T13:03:54.8122380Z 2022-08-17T13:03:54.8122898Z ---------------------------------------------------------------------- 2022-08-17T13:03:54.8123353Z Ran 1 test in 2.778s 2022-08-17T13:03:54.8123565Z 2022-08-17T13:03:54.8123685Z OK 2022-08-17T13:03:54.8123858Z 2022-08-17T13:03:54.8124603Z Generating XML reports... 2022-08-17T13:03:54.8160221Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130352.xml 2022-08-17T13:03:56.1064084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:56.1064552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:56.1067388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:56.1067799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:56.2583415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95x_e31m 2022-08-17T13:03:56.2587127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95x_e31m/_remote_module_non_scriptable.py 2022-08-17T13:03:56.5232830Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:03:56.5243769Z 2022-08-17T13:03:56.5244180Z Running tests... 2022-08-17T13:03:56.5244600Z ---------------------------------------------------------------------- 2022-08-17T13:03:56.7915387Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16726 2022-08-17T13:03:56.7944883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16727 2022-08-17T13:03:56.7974978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16728 2022-08-17T13:03:56.8006035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16729 2022-08-17T13:03:57.8223584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:57.8224380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:57.8227615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:57.8228296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:57.8441591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:57.8442255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:57.8443860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:57.8444787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:57.8505557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:57.8506245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:57.8509562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:57.8510219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:57.8518256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:03:57.8518915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:03:57.8522563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:03:57.8523249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:03:57.9827031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q6fa2pk 2022-08-17T13:03:57.9827968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q6fa2pk/_remote_module_non_scriptable.py 2022-08-17T13:03:58.0050963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kx8xa58 2022-08-17T13:03:58.0051832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kx8xa58/_remote_module_non_scriptable.py 2022-08-17T13:03:58.0104769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr520la3b 2022-08-17T13:03:58.0105840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr520la3b/_remote_module_non_scriptable.py 2022-08-17T13:03:58.0121544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1cdajd4 2022-08-17T13:03:58.0123477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1cdajd4/_remote_module_non_scriptable.py 2022-08-17T13:03:58.2463123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:03:58.2676162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:03:58.2718176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:03:58.2766334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:03:59.1068970Z ok (2.582s) 2022-08-17T13:03:59.1069193Z 2022-08-17T13:03:59.1069670Z ---------------------------------------------------------------------- 2022-08-17T13:03:59.1070112Z Ran 1 test in 2.582s 2022-08-17T13:03:59.1070310Z 2022-08-17T13:03:59.1070397Z OK 2022-08-17T13:03:59.1070546Z 2022-08-17T13:03:59.1070689Z Generating XML reports... 2022-08-17T13:03:59.1104921Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130356.xml 2022-08-17T13:04:00.4227069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:00.4227478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:00.4229354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:00.4229718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:00.5751289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiczsi7sz 2022-08-17T13:04:00.5754634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiczsi7sz/_remote_module_non_scriptable.py 2022-08-17T13:04:00.8428007Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:00.8438601Z 2022-08-17T13:04:00.8438845Z Running tests... 2022-08-17T13:04:00.8439708Z ---------------------------------------------------------------------- 2022-08-17T13:04:01.1098366Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16981 2022-08-17T13:04:01.1126242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16982 2022-08-17T13:04:01.1155056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16983 2022-08-17T13:04:01.1187732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16984 2022-08-17T13:04:02.1604618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:02.1605275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:02.1609197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:02.1609836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:02.1686252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:02.1686989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:02.1690922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:02.1691581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:02.1705876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:02.1706499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:02.1710546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:02.1711193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:02.1988948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:02.1989580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:02.1992715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:02.1993410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:02.3223811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e7b5jij 2022-08-17T13:04:02.3224679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e7b5jij/_remote_module_non_scriptable.py 2022-08-17T13:04:02.3291997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplr9phj3x 2022-08-17T13:04:02.3294413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplr9phj3x/_remote_module_non_scriptable.py 2022-08-17T13:04:02.3319812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef7ni27o 2022-08-17T13:04:02.3322245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef7ni27o/_remote_module_non_scriptable.py 2022-08-17T13:04:02.3572142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueoepvr7 2022-08-17T13:04:02.3573027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueoepvr7/_remote_module_non_scriptable.py 2022-08-17T13:04:02.5862390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:02.5934238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:02.5952160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:02.6179818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:03.3246672Z ok (2.480s) 2022-08-17T13:04:03.3246939Z 2022-08-17T13:04:03.3247278Z ---------------------------------------------------------------------- 2022-08-17T13:04:03.3247797Z Ran 1 test in 2.481s 2022-08-17T13:04:03.3247918Z 2022-08-17T13:04:03.3247982Z OK 2022-08-17T13:04:03.3248079Z 2022-08-17T13:04:03.3248174Z Generating XML reports... 2022-08-17T13:04:03.3280472Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130400.xml 2022-08-17T13:04:04.6292773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:04.6293171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:04.6295937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:04.6296315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:04.7818671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp463fxhqy 2022-08-17T13:04:04.7821498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp463fxhqy/_remote_module_non_scriptable.py 2022-08-17T13:04:05.0489250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:05.0499425Z 2022-08-17T13:04:05.0499554Z Running tests... 2022-08-17T13:04:05.0500416Z ---------------------------------------------------------------------- 2022-08-17T13:04:05.3250135Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17236 2022-08-17T13:04:05.3279090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17237 2022-08-17T13:04:05.3307268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17238 2022-08-17T13:04:05.3339106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17239 2022-08-17T13:04:06.3467311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:06.3467942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:06.3471201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:06.3471895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:06.3735895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:06.3736560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:06.3740019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:06.3740717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:06.3808606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:06.3809254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:06.3813144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:06.3813787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:06.3962058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:06.3962717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:06.3966786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:06.3967449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:06.5054616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprupcjomf 2022-08-17T13:04:06.5055667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprupcjomf/_remote_module_non_scriptable.py 2022-08-17T13:04:06.5326422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsiuixsol 2022-08-17T13:04:06.5327289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsiuixsol/_remote_module_non_scriptable.py 2022-08-17T13:04:06.5421559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaoq6fqrg 2022-08-17T13:04:06.5423058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaoq6fqrg/_remote_module_non_scriptable.py 2022-08-17T13:04:06.5534576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb_6rfjk 2022-08-17T13:04:06.5535938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb_6rfjk/_remote_module_non_scriptable.py 2022-08-17T13:04:06.7670004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:06.7931253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:06.8054363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:06.8163070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:07.5399105Z ok (2.490s) 2022-08-17T13:04:07.5399376Z 2022-08-17T13:04:07.5399707Z ---------------------------------------------------------------------- 2022-08-17T13:04:07.5399992Z Ran 1 test in 2.490s 2022-08-17T13:04:07.5400113Z 2022-08-17T13:04:07.5400178Z OK 2022-08-17T13:04:07.5400278Z 2022-08-17T13:04:07.5400364Z Generating XML reports... 2022-08-17T13:04:07.5434243Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130405.xml 2022-08-17T13:04:08.8625350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:08.8625947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:08.8648534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:08.8649531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:09.0142088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi305tiu0 2022-08-17T13:04:09.0144291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi305tiu0/_remote_module_non_scriptable.py 2022-08-17T13:04:09.2801544Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:09.2812538Z 2022-08-17T13:04:09.2812658Z Running tests... 2022-08-17T13:04:09.2813087Z ---------------------------------------------------------------------- 2022-08-17T13:04:09.5456736Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17491 2022-08-17T13:04:09.5488292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17492 2022-08-17T13:04:09.5518007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17493 2022-08-17T13:04:09.5551864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17494 2022-08-17T13:04:10.6489478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:10.6490135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:10.6493430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:10.6494068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:10.6575055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:10.6576057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:10.6580105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:10.6580731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:10.7061211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:10.7061756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:10.7065224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:10.7065820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:10.7139315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:10.7139896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:10.7143200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:10.7143904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:10.8095571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1550_ak 2022-08-17T13:04:10.8096394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1550_ak/_remote_module_non_scriptable.py 2022-08-17T13:04:10.8186367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31dhph4_ 2022-08-17T13:04:10.8187282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31dhph4_/_remote_module_non_scriptable.py 2022-08-17T13:04:10.8682680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v42rmom 2022-08-17T13:04:10.8684153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v42rmom/_remote_module_non_scriptable.py 2022-08-17T13:04:10.8751633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m9021ix 2022-08-17T13:04:10.8752808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m9021ix/_remote_module_non_scriptable.py 2022-08-17T13:04:11.0703484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:11.0816643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:11.1311964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:11.1370233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:11.5029680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:11.5130978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:11.5234397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:11.5236812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:11.5237658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:11.5238376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:11.5239166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:11.5240004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:12.0617246Z ok (2.780s) 2022-08-17T13:04:12.0617511Z 2022-08-17T13:04:12.0617941Z ---------------------------------------------------------------------- 2022-08-17T13:04:12.0618466Z Ran 1 test in 2.780s 2022-08-17T13:04:12.0618591Z 2022-08-17T13:04:12.0618657Z OK 2022-08-17T13:04:12.0618753Z 2022-08-17T13:04:12.0618846Z Generating XML reports... 2022-08-17T13:04:12.0651539Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130409.xml 2022-08-17T13:04:13.3881834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:13.3882302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:13.3884302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:13.3884664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:13.5434647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq4j3wl3 2022-08-17T13:04:13.5437776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq4j3wl3/_remote_module_non_scriptable.py 2022-08-17T13:04:13.8084379Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:13.8096009Z 2022-08-17T13:04:13.8096360Z Running tests... 2022-08-17T13:04:13.8097318Z ---------------------------------------------------------------------- 2022-08-17T13:04:14.0752956Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17738 2022-08-17T13:04:14.0780680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17739 2022-08-17T13:04:14.0809653Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17740 2022-08-17T13:04:14.0842703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17741 2022-08-17T13:04:15.0663538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:15.0664278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:15.0668178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:15.0668805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:15.0710895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:15.0711524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:15.0716153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:15.0716815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:15.1194790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:15.1195433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:15.1198602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:15.1199300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:15.1218937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:15.1219586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:15.1223847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:15.1224593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:15.2269853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvk74ybw 2022-08-17T13:04:15.2270598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvk74ybw/_remote_module_non_scriptable.py 2022-08-17T13:04:15.2312783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrmd4fvw 2022-08-17T13:04:15.2313870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrmd4fvw/_remote_module_non_scriptable.py 2022-08-17T13:04:15.2788438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vph1hc7 2022-08-17T13:04:15.2789306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vph1hc7/_remote_module_non_scriptable.py 2022-08-17T13:04:15.2834724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxnhjtux 2022-08-17T13:04:15.2835808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxnhjtux/_remote_module_non_scriptable.py 2022-08-17T13:04:15.4905904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:15.4932142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:15.5412851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:15.5484981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:15.8681781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:15.8744893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:15.8846973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:15.8850589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:15.8851893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:15.8884006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:15.8950481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:15.8951561Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:16.4905686Z ok (2.681s) 2022-08-17T13:04:16.4905878Z 2022-08-17T13:04:16.4906289Z ---------------------------------------------------------------------- 2022-08-17T13:04:16.4906543Z Ran 1 test in 2.681s 2022-08-17T13:04:16.4906665Z 2022-08-17T13:04:16.4906733Z OK 2022-08-17T13:04:16.4906829Z 2022-08-17T13:04:16.4906922Z Generating XML reports... 2022-08-17T13:04:16.4940809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130413.xml 2022-08-17T13:04:17.8088824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:17.8089232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:17.8092149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:17.8092569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:17.9609934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lf9ddr7 2022-08-17T13:04:17.9612481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lf9ddr7/_remote_module_non_scriptable.py 2022-08-17T13:04:18.2258761Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:18.2269045Z 2022-08-17T13:04:18.2269124Z Running tests... 2022-08-17T13:04:18.2270065Z ---------------------------------------------------------------------- 2022-08-17T13:04:18.4919239Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17985 2022-08-17T13:04:18.4947513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17986 2022-08-17T13:04:18.4976968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17987 2022-08-17T13:04:18.5009577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17988 2022-08-17T13:04:19.5385326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:19.5386031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:19.5386842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:19.5387461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:19.5388628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:19.5389316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:19.5391971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:19.5392940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:19.5413415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:19.5414046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:19.5418861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:19.5419543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:19.5675007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:19.5675664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:19.5678205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:19.5678911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:19.6997723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj67de79k 2022-08-17T13:04:19.6998455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxr4clwo 2022-08-17T13:04:19.6999275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj67de79k/_remote_module_non_scriptable.py 2022-08-17T13:04:19.7000024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxr4clwo/_remote_module_non_scriptable.py 2022-08-17T13:04:19.7146626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5encj34x 2022-08-17T13:04:19.7148141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5encj34x/_remote_module_non_scriptable.py 2022-08-17T13:04:19.7288451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyowa3na 2022-08-17T13:04:19.7290664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyowa3na/_remote_module_non_scriptable.py 2022-08-17T13:04:19.9644516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:19.9679840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:19.9904626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:19.9975924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:20.2826566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:20.2827508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:20.2927539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:20.2928480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:20.2929718Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:20.2931278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:20.2932095Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:20.2932876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:20.9075518Z ok (2.680s) 2022-08-17T13:04:20.9075808Z 2022-08-17T13:04:20.9076281Z ---------------------------------------------------------------------- 2022-08-17T13:04:20.9076702Z Ran 1 test in 2.681s 2022-08-17T13:04:20.9076891Z 2022-08-17T13:04:20.9076983Z OK 2022-08-17T13:04:20.9077115Z 2022-08-17T13:04:20.9077255Z Generating XML reports... 2022-08-17T13:04:20.9111717Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130418.xml 2022-08-17T13:04:22.1967013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:22.1967401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:22.1970155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:22.1970515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:22.3481425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u4hgcdg 2022-08-17T13:04:22.3484694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u4hgcdg/_remote_module_non_scriptable.py 2022-08-17T13:04:22.6104572Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:22.6115035Z 2022-08-17T13:04:22.6115184Z Running tests... 2022-08-17T13:04:22.6115793Z ---------------------------------------------------------------------- 2022-08-17T13:04:22.8775808Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18232 2022-08-17T13:04:22.8802538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18233 2022-08-17T13:04:22.8833738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18234 2022-08-17T13:04:22.8863296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18235 2022-08-17T13:04:23.8993398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:23.8993984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:23.8996762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:23.8997355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:23.9083167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:23.9083815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:23.9087276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:23.9087942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:23.9159506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:23.9160159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:23.9163078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:23.9164974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:23.9166182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:23.9166756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:23.9168306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:23.9168935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:24.0647313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojcprwli 2022-08-17T13:04:24.0648794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojcprwli/_remote_module_non_scriptable.py 2022-08-17T13:04:24.0753941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_awfan_h 2022-08-17T13:04:24.0755178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_awfan_h/_remote_module_non_scriptable.py 2022-08-17T13:04:24.0782750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6yy2g4q 2022-08-17T13:04:24.0784195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6yy2g4q/_remote_module_non_scriptable.py 2022-08-17T13:04:24.0802002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_g43xz0 2022-08-17T13:04:24.0804240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_g43xz0/_remote_module_non_scriptable.py 2022-08-17T13:04:24.3265620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:24.3395282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:24.3427936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:24.3444298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:25.0922753Z ok (2.480s) 2022-08-17T13:04:25.0923235Z 2022-08-17T13:04:25.0923696Z ---------------------------------------------------------------------- 2022-08-17T13:04:25.0923966Z Ran 1 test in 2.481s 2022-08-17T13:04:25.0924090Z 2022-08-17T13:04:25.0924141Z OK 2022-08-17T13:04:25.0924241Z 2022-08-17T13:04:25.0924336Z Generating XML reports... 2022-08-17T13:04:25.0959415Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130422.xml 2022-08-17T13:04:26.4029103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:26.4029507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:26.4031710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:26.4032080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:26.5544513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffl7ybod 2022-08-17T13:04:26.5547933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffl7ybod/_remote_module_non_scriptable.py 2022-08-17T13:04:26.8185102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:26.8196309Z 2022-08-17T13:04:26.8196783Z Running tests... 2022-08-17T13:04:26.8197357Z ---------------------------------------------------------------------- 2022-08-17T13:04:27.0876584Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18499 2022-08-17T13:04:27.0905796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18500 2022-08-17T13:04:27.0934365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18501 2022-08-17T13:04:27.0965557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18502 2022-08-17T13:04:28.1514883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:28.1515522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:28.1518023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:28.1518678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:28.1798893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:28.1799558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:28.1803927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:28.1804939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:28.1949879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:28.1950594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:28.1952904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:28.1953612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:28.2261278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:28.2262059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:28.2263818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:28.2264644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:28.3110935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73gbtm5l 2022-08-17T13:04:28.3111812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73gbtm5l/_remote_module_non_scriptable.py 2022-08-17T13:04:28.3397752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjxyjqu8 2022-08-17T13:04:28.3398621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjxyjqu8/_remote_module_non_scriptable.py 2022-08-17T13:04:28.3544173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o2yng20 2022-08-17T13:04:28.3544963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o2yng20/_remote_module_non_scriptable.py 2022-08-17T13:04:28.3841704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaialpfbc 2022-08-17T13:04:28.3842558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaialpfbc/_remote_module_non_scriptable.py 2022-08-17T13:04:28.5749935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:28.6002651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:28.6122065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:28.6438207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:28.8938726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:28.9037228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:28.9138461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:28.9139762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:28.9141451Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:28.9142958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:28.9144498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:28.9149619Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:29.6032770Z ok (2.783s) 2022-08-17T13:04:29.6032999Z 2022-08-17T13:04:29.6033505Z ---------------------------------------------------------------------- 2022-08-17T13:04:29.6033909Z Ran 1 test in 2.783s 2022-08-17T13:04:29.6034102Z 2022-08-17T13:04:29.6034203Z OK 2022-08-17T13:04:29.6034368Z 2022-08-17T13:04:29.6034517Z Generating XML reports... 2022-08-17T13:04:29.6068624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130426.xml 2022-08-17T13:04:30.9143284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:30.9143665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:30.9151550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:30.9151959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:31.0664262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqp2w9or 2022-08-17T13:04:31.0667325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqp2w9or/_remote_module_non_scriptable.py 2022-08-17T13:04:31.3333373Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:31.3343846Z 2022-08-17T13:04:31.3344213Z Running tests... 2022-08-17T13:04:31.3344829Z ---------------------------------------------------------------------- 2022-08-17T13:04:31.5991216Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18746 2022-08-17T13:04:31.6019565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18747 2022-08-17T13:04:31.6048371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18748 2022-08-17T13:04:31.6078356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18749 2022-08-17T13:04:32.6469260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:32.6469873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:32.6472965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:32.6473605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:32.6531997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:32.6532663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:32.6535823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:32.6536476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:32.6542362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:32.6543251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:32.6547780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:32.6548432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:32.6558839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:32.6559467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:32.6563847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:32.6564518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:32.8100335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvhfrtcy 2022-08-17T13:04:32.8101790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvhfrtcy/_remote_module_non_scriptable.py 2022-08-17T13:04:32.8144824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw4kkq07 2022-08-17T13:04:32.8146218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw4kkq07/_remote_module_non_scriptable.py 2022-08-17T13:04:32.8168024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f9w1lqc 2022-08-17T13:04:32.8170025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f9w1lqc/_remote_module_non_scriptable.py 2022-08-17T13:04:32.8177274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55fek18f 2022-08-17T13:04:32.8179673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55fek18f/_remote_module_non_scriptable.py 2022-08-17T13:04:33.0763420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:33.0790188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:33.0822573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:33.0834590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:33.3624139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:33.3625767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:33.3731859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:33.3732984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:33.3734340Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:33.3735570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:33.3831563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:33.3834253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:34.1144392Z ok (2.780s) 2022-08-17T13:04:34.1144568Z 2022-08-17T13:04:34.1145448Z ---------------------------------------------------------------------- 2022-08-17T13:04:34.1145752Z Ran 1 test in 2.780s 2022-08-17T13:04:34.1145864Z 2022-08-17T13:04:34.1145928Z OK 2022-08-17T13:04:34.1146030Z 2022-08-17T13:04:34.1146125Z Generating XML reports... 2022-08-17T13:04:34.1182129Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130431.xml 2022-08-17T13:04:35.3848600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:35.3848993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:35.3851903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:35.3852267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:35.5362046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vigld5n 2022-08-17T13:04:35.5365266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vigld5n/_remote_module_non_scriptable.py 2022-08-17T13:04:35.8040837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:35.8052186Z 2022-08-17T13:04:35.8052308Z Running tests... 2022-08-17T13:04:35.8052903Z ---------------------------------------------------------------------- 2022-08-17T13:04:36.0688383Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18993 2022-08-17T13:04:36.0716888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18994 2022-08-17T13:04:36.0745033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18995 2022-08-17T13:04:36.0776964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18996 2022-08-17T13:04:37.1109749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:37.1110377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:37.1113678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:37.1114344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:37.1232202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:37.1232875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:37.1237189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:37.1237811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:37.1238600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:37.1239177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:37.1242406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:37.1243057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:37.1453687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:37.1454347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:37.1457535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:37.1458236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:37.2688305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fa21akx 2022-08-17T13:04:37.2689136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fa21akx/_remote_module_non_scriptable.py 2022-08-17T13:04:37.2821074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yg450ed 2022-08-17T13:04:37.2822528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yg450ed/_remote_module_non_scriptable.py 2022-08-17T13:04:37.2844352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgooo3h_9 2022-08-17T13:04:37.2846521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgooo3h_9/_remote_module_non_scriptable.py 2022-08-17T13:04:37.3026993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ejpc_mn 2022-08-17T13:04:37.3039529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ejpc_mn/_remote_module_non_scriptable.py 2022-08-17T13:04:37.5263043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:37.5416312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:37.5479428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:37.5800778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:37.8559744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:37.8658990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:37.8761722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:37.8763236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:37.8764839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:37.8766471Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:37.8768391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:37.8769909Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:38.5841994Z ok (2.779s) 2022-08-17T13:04:38.5842258Z 2022-08-17T13:04:38.5842654Z ---------------------------------------------------------------------- 2022-08-17T13:04:38.5842922Z Ran 1 test in 2.779s 2022-08-17T13:04:38.5843046Z 2022-08-17T13:04:38.5843109Z OK 2022-08-17T13:04:38.5843204Z 2022-08-17T13:04:38.5843303Z Generating XML reports... 2022-08-17T13:04:38.5897778Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130435.xml 2022-08-17T13:04:39.8719728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:39.8720138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:39.8722664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:39.8723035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:40.0225642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lca6hef 2022-08-17T13:04:40.0228959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lca6hef/_remote_module_non_scriptable.py 2022-08-17T13:04:40.2883093Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:40.2894692Z 2022-08-17T13:04:40.2894836Z Running tests... 2022-08-17T13:04:40.2895259Z ---------------------------------------------------------------------- 2022-08-17T13:04:40.5521688Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19240 2022-08-17T13:04:40.5551143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19241 2022-08-17T13:04:40.5580421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19242 2022-08-17T13:04:40.5611664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19243 2022-08-17T13:04:41.6039166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:41.6039817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:41.6042086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:41.6042733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:41.6646907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:41.6647473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:41.6650181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:41.6650756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:41.6866171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:41.6866785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:41.6870156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:41.6870816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:41.7249483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:41.7250125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:41.7252947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:41.7253909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:41.7683362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr0d9grv 2022-08-17T13:04:41.7684185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr0d9grv/_remote_module_non_scriptable.py 2022-08-17T13:04:41.8307812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflga5yny 2022-08-17T13:04:41.8308579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflga5yny/_remote_module_non_scriptable.py 2022-08-17T13:04:41.8476161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7osamvz8 2022-08-17T13:04:41.8477009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7osamvz8/_remote_module_non_scriptable.py 2022-08-17T13:04:41.8857072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98r5sxz4 2022-08-17T13:04:41.8857929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98r5sxz4/_remote_module_non_scriptable.py 2022-08-17T13:04:42.0333149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:42.0949064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:42.1107085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:42.1510983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:42.4279124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:42.4380431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:42.4381336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:42.4387067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:42.4388472Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:42.4486179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:42.4487489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:42.4488741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:43.0678525Z ok (2.778s) 2022-08-17T13:04:43.0678750Z 2022-08-17T13:04:43.0679281Z ---------------------------------------------------------------------- 2022-08-17T13:04:43.0679690Z Ran 1 test in 2.778s 2022-08-17T13:04:43.0679799Z 2022-08-17T13:04:43.0679866Z OK 2022-08-17T13:04:43.0679962Z 2022-08-17T13:04:43.0680060Z Generating XML reports... 2022-08-17T13:04:43.0713279Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130440.xml 2022-08-17T13:04:44.3531358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:44.3532054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:44.3535057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:44.3535669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:44.5038693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j2uw_kl 2022-08-17T13:04:44.5041598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j2uw_kl/_remote_module_non_scriptable.py 2022-08-17T13:04:44.7676342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:44.7687075Z 2022-08-17T13:04:44.7687403Z Running tests... 2022-08-17T13:04:44.7688089Z ---------------------------------------------------------------------- 2022-08-17T13:04:45.0324033Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19487 2022-08-17T13:04:45.0353673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19488 2022-08-17T13:04:45.0383581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19489 2022-08-17T13:04:45.0415395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19490 2022-08-17T13:04:46.0513265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:46.0513849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:46.0516463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:46.0517057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:46.0683425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:46.0684087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:46.0687084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:46.0687737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:46.0879838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:46.0880479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:46.0883701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:46.0884368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:46.0930371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:46.0931047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:46.0934968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:46.0935617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:46.2117494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp9jl8vf 2022-08-17T13:04:46.2118326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp9jl8vf/_remote_module_non_scriptable.py 2022-08-17T13:04:46.2312980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbg7omb0 2022-08-17T13:04:46.2314233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbg7omb0/_remote_module_non_scriptable.py 2022-08-17T13:04:46.2491266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5r58c_f 2022-08-17T13:04:46.2492078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5r58c_f/_remote_module_non_scriptable.py 2022-08-17T13:04:46.2536384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraed8l3r 2022-08-17T13:04:46.2537465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraed8l3r/_remote_module_non_scriptable.py 2022-08-17T13:04:46.4747268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:46.4951759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:46.5131446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:46.5187264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:46.7642642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:46.7743196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:46.7744153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:46.7747693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:46.7748945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:46.7750509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:46.7845720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:46.7853264Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:47.4478603Z ok (2.679s) 2022-08-17T13:04:47.4478861Z 2022-08-17T13:04:47.4479211Z ---------------------------------------------------------------------- 2022-08-17T13:04:47.4479479Z Ran 1 test in 2.679s 2022-08-17T13:04:47.4479595Z 2022-08-17T13:04:47.4479661Z OK 2022-08-17T13:04:47.4479776Z 2022-08-17T13:04:47.4479877Z Generating XML reports... 2022-08-17T13:04:47.4512495Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130444.xml 2022-08-17T13:04:48.7643817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:48.7646180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:48.7646904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:48.7647530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:48.9184644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4doorcan 2022-08-17T13:04:48.9187686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4doorcan/_remote_module_non_scriptable.py 2022-08-17T13:04:49.1855514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:49.1866089Z 2022-08-17T13:04:49.1866237Z Running tests... 2022-08-17T13:04:49.1866736Z ---------------------------------------------------------------------- 2022-08-17T13:04:49.4509417Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19734 2022-08-17T13:04:49.4538443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19735 2022-08-17T13:04:49.4567776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19736 2022-08-17T13:04:49.4598394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19737 2022-08-17T13:04:50.4954491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:50.4955480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:50.4957016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:50.4957657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:50.5080355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:50.5081045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:50.5085183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:50.5086036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:50.5199182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:50.5199820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:50.5203118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:50.5203773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:50.5377813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:50.5378535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:50.5381195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:50.5381916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:50.6539954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij10cumi 2022-08-17T13:04:50.6540803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij10cumi/_remote_module_non_scriptable.py 2022-08-17T13:04:50.6689708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q6wws8w 2022-08-17T13:04:50.6690717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q6wws8w/_remote_module_non_scriptable.py 2022-08-17T13:04:50.6799720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyruvzf6 2022-08-17T13:04:50.6801052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyruvzf6/_remote_module_non_scriptable.py 2022-08-17T13:04:50.6986811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyno9c1gy 2022-08-17T13:04:50.6987611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyno9c1gy/_remote_module_non_scriptable.py 2022-08-17T13:04:50.9145650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:50.9329491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:50.9430828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:50.9668751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:51.2500853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:51.2501778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:51.2602826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:51.2603741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:51.2605113Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:51.2606812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:51.2608483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:51.2609718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:51.9663712Z ok (2.779s) 2022-08-17T13:04:51.9664043Z 2022-08-17T13:04:51.9664561Z ---------------------------------------------------------------------- 2022-08-17T13:04:51.9664899Z Ran 1 test in 2.780s 2022-08-17T13:04:51.9665021Z 2022-08-17T13:04:51.9665087Z OK 2022-08-17T13:04:51.9665183Z 2022-08-17T13:04:51.9665277Z Generating XML reports... 2022-08-17T13:04:51.9697861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130449.xml 2022-08-17T13:04:53.2567306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:53.2567719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:53.2570653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:53.2571062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:53.4087750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1_vs8lg 2022-08-17T13:04:53.4091477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1_vs8lg/_remote_module_non_scriptable.py 2022-08-17T13:04:53.6752915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:53.6763735Z 2022-08-17T13:04:53.6763980Z Running tests... 2022-08-17T13:04:53.6764396Z ---------------------------------------------------------------------- 2022-08-17T13:04:53.9396349Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19981 2022-08-17T13:04:53.9424662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19982 2022-08-17T13:04:53.9453806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19983 2022-08-17T13:04:53.9486628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19984 2022-08-17T13:04:54.9862930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:54.9863562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:54.9866362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:54.9867296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:55.0248242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:55.0248912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:55.0251768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:55.0252425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:55.0381608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:55.0382217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:55.0385547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:55.0386263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:55.0593811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:55.0594509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:55.0597285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:55.0597945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:55.1429065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a9swyjg 2022-08-17T13:04:55.1429933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a9swyjg/_remote_module_non_scriptable.py 2022-08-17T13:04:55.1836698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp637plaw4 2022-08-17T13:04:55.1837978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp637plaw4/_remote_module_non_scriptable.py 2022-08-17T13:04:55.1980375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjh4v2dkl 2022-08-17T13:04:55.1981315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjh4v2dkl/_remote_module_non_scriptable.py 2022-08-17T13:04:55.2165619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9s3qy_aq 2022-08-17T13:04:55.2166582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9s3qy_aq/_remote_module_non_scriptable.py 2022-08-17T13:04:55.4037973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:04:55.4485950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:55.4606005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:55.4992356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:55.7398882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:04:55.7495904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:04:55.7597787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:04:55.7599128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:55.7601157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:55.7601944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:04:55.7602895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:55.7609686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:04:56.4552421Z ok (2.779s) 2022-08-17T13:04:56.4552928Z 2022-08-17T13:04:56.4553570Z ---------------------------------------------------------------------- 2022-08-17T13:04:56.4554020Z Ran 1 test in 2.779s 2022-08-17T13:04:56.4554223Z 2022-08-17T13:04:56.4555274Z OK 2022-08-17T13:04:56.4555427Z 2022-08-17T13:04:56.4555677Z Generating XML reports... 2022-08-17T13:04:56.4588609Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130453.xml 2022-08-17T13:04:57.7646528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:57.7646928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:57.7649673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:57.7650066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:57.9171041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpawha37 2022-08-17T13:04:57.9174499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpawha37/_remote_module_non_scriptable.py 2022-08-17T13:04:58.1808610Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:04:58.1819780Z 2022-08-17T13:04:58.1819934Z Running tests... 2022-08-17T13:04:58.1820998Z ---------------------------------------------------------------------- 2022-08-17T13:04:58.4491763Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20228 2022-08-17T13:04:58.4518717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20229 2022-08-17T13:04:58.4551084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20230 2022-08-17T13:04:58.4581152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20231 2022-08-17T13:04:59.4781440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:59.4782050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:59.4784774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:59.4785354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:59.4852374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:59.4853088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:59.4857344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:59.4857988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:59.5111899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:59.5112608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:59.5115299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:59.5115966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:59.5530899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:04:59.5531567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:04:59.5534464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:04:59.5535397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:04:59.6428270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7vjmc9y 2022-08-17T13:04:59.6429508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7vjmc9y/_remote_module_non_scriptable.py 2022-08-17T13:04:59.6458696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpll0avend 2022-08-17T13:04:59.6460630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpll0avend/_remote_module_non_scriptable.py 2022-08-17T13:04:59.6710886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlsld65l 2022-08-17T13:04:59.6711749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlsld65l/_remote_module_non_scriptable.py 2022-08-17T13:04:59.7113530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejz6faf2 2022-08-17T13:04:59.7114465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejz6faf2/_remote_module_non_scriptable.py 2022-08-17T13:04:59.9085049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:04:59.9089290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:04:59.9358930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:04:59.9774024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:00.2666710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:05:00.2767712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:05:00.2771288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:05:00.2772667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:00.2773633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:05:00.2774821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:00.2870446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:00.2871733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:00.9648886Z ok (2.783s) 2022-08-17T13:05:00.9649139Z 2022-08-17T13:05:00.9649673Z ---------------------------------------------------------------------- 2022-08-17T13:05:00.9649974Z Ran 1 test in 2.783s 2022-08-17T13:05:00.9650099Z 2022-08-17T13:05:00.9650164Z OK 2022-08-17T13:05:00.9650263Z 2022-08-17T13:05:00.9650374Z Generating XML reports... 2022-08-17T13:05:00.9684790Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130458.xml 2022-08-17T13:05:02.2947184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:02.2947565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:02.2950280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:02.2950766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:02.4473515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mtl8pp0 2022-08-17T13:05:02.4476199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mtl8pp0/_remote_module_non_scriptable.py 2022-08-17T13:05:02.7139996Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:02.7150370Z 2022-08-17T13:05:02.7150525Z Running tests... 2022-08-17T13:05:02.7151597Z ---------------------------------------------------------------------- 2022-08-17T13:05:02.9809384Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20475 2022-08-17T13:05:02.9839940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20476 2022-08-17T13:05:02.9870862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20477 2022-08-17T13:05:02.9904190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20478 2022-08-17T13:05:04.0165204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:04.0166126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:04.0168521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:04.0169175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:04.0189484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:04.0190132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:04.0195087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:04.0195756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:04.0219295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:04.0219888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:04.0224346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:04.0224984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:04.0430920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:04.0431581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:04.0432730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:04.0433358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:04.1787054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu4kejjt 2022-08-17T13:05:04.1787839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu4kejjt/_remote_module_non_scriptable.py 2022-08-17T13:05:04.1789656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhamn9a9 2022-08-17T13:05:04.1792518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhamn9a9/_remote_module_non_scriptable.py 2022-08-17T13:05:04.1815451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76n7l6yp 2022-08-17T13:05:04.1816941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76n7l6yp/_remote_module_non_scriptable.py 2022-08-17T13:05:04.2001555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinr8ithy 2022-08-17T13:05:04.2002494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinr8ithy/_remote_module_non_scriptable.py 2022-08-17T13:05:04.4423035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:04.4424167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:04.4447697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:04.4614534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:05.2966992Z ok (2.581s) 2022-08-17T13:05:05.2967254Z 2022-08-17T13:05:05.2967764Z ---------------------------------------------------------------------- 2022-08-17T13:05:05.2968099Z Ran 1 test in 2.582s 2022-08-17T13:05:05.2968209Z 2022-08-17T13:05:05.2968297Z OK 2022-08-17T13:05:05.2968396Z 2022-08-17T13:05:05.2968492Z Generating XML reports... 2022-08-17T13:05:05.3001365Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130502.xml 2022-08-17T13:05:06.6190546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:06.6190931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:06.6193298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:06.6193697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:06.7712104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpye5p4gab 2022-08-17T13:05:06.7715936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpye5p4gab/_remote_module_non_scriptable.py 2022-08-17T13:05:07.0346059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:07.0357029Z 2022-08-17T13:05:07.0357142Z Running tests... 2022-08-17T13:05:07.0357880Z ---------------------------------------------------------------------- 2022-08-17T13:05:07.2972208Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20730 2022-08-17T13:05:07.3000578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20731 2022-08-17T13:05:07.3029732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20732 2022-08-17T13:05:07.3060532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20733 2022-08-17T13:05:08.3190155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:08.3190874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:08.3193765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:08.3194424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:08.3837444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:08.3838075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:08.3841094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:08.3841945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:08.3990542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:08.3991154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:08.3993881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:08.3995166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:08.4007553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:08.4008196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:08.4011605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:08.4012516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:08.4789589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y9pcz_9 2022-08-17T13:05:08.4790707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y9pcz_9/_remote_module_non_scriptable.py 2022-08-17T13:05:08.5428243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpmtshbk 2022-08-17T13:05:08.5429035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpmtshbk/_remote_module_non_scriptable.py 2022-08-17T13:05:08.5576077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c2o9e90 2022-08-17T13:05:08.5577134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c2o9e90/_remote_module_non_scriptable.py 2022-08-17T13:05:08.5580935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_5g79s_ 2022-08-17T13:05:08.5583805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_5g79s_/_remote_module_non_scriptable.py 2022-08-17T13:05:08.7383824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:08.8036088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:08.8168431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:08.8169649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:10.5140822Z ok (3.478s) 2022-08-17T13:05:10.5141084Z 2022-08-17T13:05:10.5141621Z ---------------------------------------------------------------------- 2022-08-17T13:05:10.5141936Z Ran 1 test in 3.478s 2022-08-17T13:05:10.5142061Z 2022-08-17T13:05:10.5142129Z OK 2022-08-17T13:05:10.5142227Z 2022-08-17T13:05:10.5142327Z Generating XML reports... 2022-08-17T13:05:10.5176444Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130507.xml 2022-08-17T13:05:11.8355645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:11.8356280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:11.8358657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:11.8359262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:11.9848418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqmkinrs 2022-08-17T13:05:11.9851740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqmkinrs/_remote_module_non_scriptable.py 2022-08-17T13:05:12.2570024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:12.2580755Z 2022-08-17T13:05:12.2581217Z Running tests... 2022-08-17T13:05:12.2581669Z ---------------------------------------------------------------------- 2022-08-17T13:05:12.5194817Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20985 2022-08-17T13:05:12.5224490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20986 2022-08-17T13:05:12.5253040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20987 2022-08-17T13:05:12.5284285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20988 2022-08-17T13:05:13.5913054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:13.5913705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:13.5915961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:13.5916876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:13.6037002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:13.6037632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:13.6040423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:13.6041066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:13.6198122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:13.6198796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:13.6201271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:13.6201899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:13.6278429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:13.6279107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:13.6282694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:13.6283381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:13.7485390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8kc76zs 2022-08-17T13:05:13.7486438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8kc76zs/_remote_module_non_scriptable.py 2022-08-17T13:05:13.7634178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64xeqtj5 2022-08-17T13:05:13.7635244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64xeqtj5/_remote_module_non_scriptable.py 2022-08-17T13:05:13.7793277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5vy7izt 2022-08-17T13:05:13.7794475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5vy7izt/_remote_module_non_scriptable.py 2022-08-17T13:05:13.7854107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjw81ts3 2022-08-17T13:05:13.7855747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjw81ts3/_remote_module_non_scriptable.py 2022-08-17T13:05:14.0058084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:14.0244724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:14.0408667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:14.0458854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:14.8345237Z ok (2.576s) 2022-08-17T13:05:14.8345518Z 2022-08-17T13:05:14.8346025Z ---------------------------------------------------------------------- 2022-08-17T13:05:14.8346534Z Ran 1 test in 2.576s 2022-08-17T13:05:14.8346695Z 2022-08-17T13:05:14.8346759Z OK 2022-08-17T13:05:14.8346855Z 2022-08-17T13:05:14.8346952Z Generating XML reports... 2022-08-17T13:05:14.8381360Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130512.xml 2022-08-17T13:05:16.1418204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:16.1418637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:16.1420888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:16.1421355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:16.2945577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhthd9xt 2022-08-17T13:05:16.2947822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhthd9xt/_remote_module_non_scriptable.py 2022-08-17T13:05:16.5609032Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:16.5619973Z 2022-08-17T13:05:16.5620068Z Running tests... 2022-08-17T13:05:16.5621256Z ---------------------------------------------------------------------- 2022-08-17T13:05:16.8232230Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21240 2022-08-17T13:05:16.8260079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21241 2022-08-17T13:05:16.8289383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21242 2022-08-17T13:05:16.8321587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21243 2022-08-17T13:05:17.8235605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:17.8236258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:17.8239740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:17.8240441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:17.8670044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:17.8670751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:17.8673122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:17.8673823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:17.9812667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g5fb9v5 2022-08-17T13:05:17.9813495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g5fb9v5/_remote_module_non_scriptable.py 2022-08-17T13:05:18.0157964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:18.0158660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:18.0160354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:18.0161411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:18.0163266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:18.0163879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:18.0168592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:18.0169569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:18.0264372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw4y1edq 2022-08-17T13:05:18.0266267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw4y1edq/_remote_module_non_scriptable.py 2022-08-17T13:05:18.1725784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi05o668f 2022-08-17T13:05:18.1727410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi05o668f/_remote_module_non_scriptable.py 2022-08-17T13:05:18.1728330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk_l7kyz 2022-08-17T13:05:18.1730920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk_l7kyz/_remote_module_non_scriptable.py 2022-08-17T13:05:18.2429998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:18.2933173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:18.4316950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:18.4332931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:18.7002278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:05:18.7101302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:05:18.7205596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:05:18.7206441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:05:18.7207690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:18.7208970Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:18.7210470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:18.7211714Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:19.3382913Z ok (2.776s) 2022-08-17T13:05:19.3383181Z 2022-08-17T13:05:19.3383713Z ---------------------------------------------------------------------- 2022-08-17T13:05:19.3384089Z Ran 1 test in 2.776s 2022-08-17T13:05:19.3384215Z 2022-08-17T13:05:19.3384280Z OK 2022-08-17T13:05:19.3384376Z 2022-08-17T13:05:19.3384477Z Generating XML reports... 2022-08-17T13:05:19.3417544Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130516.xml 2022-08-17T13:05:20.6368727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:20.6369130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:20.6372367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:20.6372818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:20.7881051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq02hncsr 2022-08-17T13:05:20.7884758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq02hncsr/_remote_module_non_scriptable.py 2022-08-17T13:05:21.0530397Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:21.0542477Z 2022-08-17T13:05:21.0542633Z Running tests... 2022-08-17T13:05:21.0543116Z ---------------------------------------------------------------------- 2022-08-17T13:05:21.3220672Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21487 2022-08-17T13:05:21.3251582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21488 2022-08-17T13:05:21.3279439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21489 2022-08-17T13:05:21.3311329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21490 2022-08-17T13:05:22.3548738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:22.3549370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:22.3552147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:22.3553135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:22.3576340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:22.3576948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:22.3580869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:22.3581543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:22.3691123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:22.3691768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:22.3695105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:22.3695758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:22.3779012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:22.3779692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:22.3783379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:22.3784116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:22.5137851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lcn67jw 2022-08-17T13:05:22.5139407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lcn67jw/_remote_module_non_scriptable.py 2022-08-17T13:05:22.5192130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo27nwaz 2022-08-17T13:05:22.5193367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo27nwaz/_remote_module_non_scriptable.py 2022-08-17T13:05:22.5285644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzatr7z1 2022-08-17T13:05:22.5287783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzatr7z1/_remote_module_non_scriptable.py 2022-08-17T13:05:22.5601891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42kcjjkc 2022-08-17T13:05:22.5602932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42kcjjkc/_remote_module_non_scriptable.py 2022-08-17T13:05:22.7715918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:22.7800004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:22.7913093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:22.8211986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:23.0661618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:05:23.0662563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:05:23.0762332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:05:23.0763562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:05:23.0764896Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:23.0766285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:23.0767529Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:23.0768781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:23.7375367Z ok (2.683s) 2022-08-17T13:05:23.7375611Z 2022-08-17T13:05:23.7375959Z ---------------------------------------------------------------------- 2022-08-17T13:05:23.7376225Z Ran 1 test in 2.683s 2022-08-17T13:05:23.7376345Z 2022-08-17T13:05:23.7376410Z OK 2022-08-17T13:05:23.7376517Z 2022-08-17T13:05:23.7376613Z Generating XML reports... 2022-08-17T13:05:23.7410061Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130521.xml 2022-08-17T13:05:25.0355493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:25.0355898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:25.0358343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:25.0358743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:25.1842727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hjx8fjp 2022-08-17T13:05:25.1845774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hjx8fjp/_remote_module_non_scriptable.py 2022-08-17T13:05:25.4455392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:25.4465976Z 2022-08-17T13:05:25.4466126Z Running tests... 2022-08-17T13:05:25.4466775Z ---------------------------------------------------------------------- 2022-08-17T13:05:25.7072667Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21734 2022-08-17T13:05:25.7100979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21735 2022-08-17T13:05:25.7129875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21736 2022-08-17T13:05:25.7160695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21737 2022-08-17T13:05:26.7031098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:26.7031802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:26.7034561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:26.7035170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:26.7106327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:26.7106957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:26.7110002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:26.7110654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:26.7477627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:26.7478305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:26.7481970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:26.7482612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:26.7708323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:26.7708992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:26.7711865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:26.7712565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:26.8625218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd47h6nqa 2022-08-17T13:05:26.8626455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd47h6nqa/_remote_module_non_scriptable.py 2022-08-17T13:05:26.8684236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuirk0pm7 2022-08-17T13:05:26.8685530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuirk0pm7/_remote_module_non_scriptable.py 2022-08-17T13:05:26.9074311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5errw92m 2022-08-17T13:05:26.9075025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5errw92m/_remote_module_non_scriptable.py 2022-08-17T13:05:26.9309353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nvao39z 2022-08-17T13:05:26.9310361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nvao39z/_remote_module_non_scriptable.py 2022-08-17T13:05:27.1248161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:27.1329681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:27.1692187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:27.1944415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:28.0221828Z ok (2.575s) 2022-08-17T13:05:28.0222004Z 2022-08-17T13:05:28.0222391Z ---------------------------------------------------------------------- 2022-08-17T13:05:28.0222670Z Ran 1 test in 2.576s 2022-08-17T13:05:28.0222793Z 2022-08-17T13:05:28.0222884Z OK 2022-08-17T13:05:28.0223021Z 2022-08-17T13:05:28.0223120Z Generating XML reports... 2022-08-17T13:05:28.0256476Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130525.xml 2022-08-17T13:05:29.3166519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:29.3166908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:29.3169612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:29.3169989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:29.4721256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc5ulwxx 2022-08-17T13:05:29.4724674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc5ulwxx/_remote_module_non_scriptable.py 2022-08-17T13:05:29.7405695Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:29.7416898Z 2022-08-17T13:05:29.7416983Z Running tests... 2022-08-17T13:05:29.7417484Z ---------------------------------------------------------------------- 2022-08-17T13:05:30.0088261Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21981 2022-08-17T13:05:30.0117623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21982 2022-08-17T13:05:30.0148111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21983 2022-08-17T13:05:30.0180764Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21984 2022-08-17T13:05:31.0417837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:31.0418440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:31.0421446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:31.0422058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:31.0498186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:31.0498881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:31.0502775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:31.0503419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:31.0566421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:31.0567057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:31.0570878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:31.0571553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:31.0761190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:31.0761861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:31.0764716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:31.0765397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:31.2031769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2e4wovs 2022-08-17T13:05:31.2032859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2e4wovs/_remote_module_non_scriptable.py 2022-08-17T13:05:31.2100825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbj0atdu 2022-08-17T13:05:31.2101945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbj0atdu/_remote_module_non_scriptable.py 2022-08-17T13:05:31.2162196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g_jj2v8 2022-08-17T13:05:31.2163639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g_jj2v8/_remote_module_non_scriptable.py 2022-08-17T13:05:31.2336442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jzkrm21 2022-08-17T13:05:31.2337204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jzkrm21/_remote_module_non_scriptable.py 2022-08-17T13:05:31.4686067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:31.4725798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:31.4794681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:31.4946332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:32.7249185Z ok (2.983s) 2022-08-17T13:05:32.7249442Z 2022-08-17T13:05:32.7250008Z ---------------------------------------------------------------------- 2022-08-17T13:05:32.7250282Z Ran 1 test in 2.983s 2022-08-17T13:05:32.7250404Z 2022-08-17T13:05:32.7250468Z OK 2022-08-17T13:05:32.7250564Z 2022-08-17T13:05:32.7250660Z Generating XML reports... 2022-08-17T13:05:32.7284259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130529.xml 2022-08-17T13:05:34.0333153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:34.0333608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:34.0336187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:34.0336563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:34.1843836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l2exnfz 2022-08-17T13:05:34.1847241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l2exnfz/_remote_module_non_scriptable.py 2022-08-17T13:05:34.4499443Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:34.4510276Z 2022-08-17T13:05:34.4510439Z Running tests... 2022-08-17T13:05:34.4510933Z ---------------------------------------------------------------------- 2022-08-17T13:05:34.7154340Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22228 2022-08-17T13:05:34.7182613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22229 2022-08-17T13:05:34.7211393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22230 2022-08-17T13:05:34.7245738Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22231 2022-08-17T13:05:35.8157403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:35.8158043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:35.8161269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:35.8161951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:35.8203138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:35.8203750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:35.8207444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:35.8208063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:35.8389998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:35.8390716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:35.8394272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:35.8394946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:35.8688144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:35.8688716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:35.8692105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:35.8692768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:35.9768606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g_5zse7 2022-08-17T13:05:35.9769866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g_5zse7/_remote_module_non_scriptable.py 2022-08-17T13:05:35.9797527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6ho01_r 2022-08-17T13:05:35.9799077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6ho01_r/_remote_module_non_scriptable.py 2022-08-17T13:05:35.9977155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdkndb4r 2022-08-17T13:05:35.9978010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdkndb4r/_remote_module_non_scriptable.py 2022-08-17T13:05:36.0286442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq57lu1_i 2022-08-17T13:05:36.0287172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq57lu1_i/_remote_module_non_scriptable.py 2022-08-17T13:05:36.2428751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:36.2605497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:36.2628866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:36.2939919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:37.3312505Z ok (2.880s) 2022-08-17T13:05:37.3312715Z 2022-08-17T13:05:37.3313260Z ---------------------------------------------------------------------- 2022-08-17T13:05:37.3313728Z Ran 1 test in 2.880s 2022-08-17T13:05:37.3313934Z 2022-08-17T13:05:37.3314030Z OK 2022-08-17T13:05:37.3314215Z 2022-08-17T13:05:37.3314392Z Generating XML reports... 2022-08-17T13:05:37.3347666Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130534.xml 2022-08-17T13:05:38.6395525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:38.6395932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:38.6398149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:38.6398571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:38.7887288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs974dcs 2022-08-17T13:05:38.7889863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs974dcs/_remote_module_non_scriptable.py 2022-08-17T13:05:39.0513214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:39.0523844Z 2022-08-17T13:05:39.0523952Z Running tests... 2022-08-17T13:05:39.0524417Z ---------------------------------------------------------------------- 2022-08-17T13:05:39.3223383Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22495 2022-08-17T13:05:39.3252613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22496 2022-08-17T13:05:39.3281584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22497 2022-08-17T13:05:39.3312865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22498 2022-08-17T13:05:40.3559789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:40.3560499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:40.3563261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:40.3563915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:40.3911541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:40.3912215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:40.3914464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:40.3915122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:40.3953488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:40.3954104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:40.3957840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:40.3958366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:40.4390307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:40.4390921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:40.4394290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:40.4394976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:40.5148492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4itkc_a 2022-08-17T13:05:40.5149584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4itkc_a/_remote_module_non_scriptable.py 2022-08-17T13:05:40.5520966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p5w21e9 2022-08-17T13:05:40.5521733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p5w21e9/_remote_module_non_scriptable.py 2022-08-17T13:05:40.5540450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzrtg8yw 2022-08-17T13:05:40.5542329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzrtg8yw/_remote_module_non_scriptable.py 2022-08-17T13:05:40.5965414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxxci1xo 2022-08-17T13:05:40.5966454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxxci1xo/_remote_module_non_scriptable.py 2022-08-17T13:05:40.7764014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:40.8149210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:40.8149908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:40.8568469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:41.6375550Z ok (2.585s) 2022-08-17T13:05:41.6375839Z 2022-08-17T13:05:41.6376386Z ---------------------------------------------------------------------- 2022-08-17T13:05:41.6376721Z Ran 1 test in 2.585s 2022-08-17T13:05:41.6376842Z 2022-08-17T13:05:41.6376924Z OK 2022-08-17T13:05:41.6377007Z 2022-08-17T13:05:41.6377105Z Generating XML reports... 2022-08-17T13:05:41.6413340Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130539.xml 2022-08-17T13:05:42.9512120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:42.9512518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:42.9514880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:42.9515327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:43.1016486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbaprs1_e 2022-08-17T13:05:43.1019636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbaprs1_e/_remote_module_non_scriptable.py 2022-08-17T13:05:43.3629371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:43.3639840Z 2022-08-17T13:05:43.3639963Z Running tests... 2022-08-17T13:05:43.3640496Z ---------------------------------------------------------------------- 2022-08-17T13:05:43.6294353Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22730 2022-08-17T13:05:43.6324576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22731 2022-08-17T13:05:43.6353984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22732 2022-08-17T13:05:43.6384519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22733 2022-08-17T13:05:44.6533651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:44.6534245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:44.6536738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:44.6537321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:44.6794174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:44.6794826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:44.6809848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:44.6810614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:44.6942875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:44.6943522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:44.6946993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:44.6947634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:44.7069490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:44.7070171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:44.7073131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:44.7073778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:44.8145363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps83kx3ww 2022-08-17T13:05:44.8146214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps83kx3ww/_remote_module_non_scriptable.py 2022-08-17T13:05:44.8400632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03zkne2h 2022-08-17T13:05:44.8401423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03zkne2h/_remote_module_non_scriptable.py 2022-08-17T13:05:44.8547256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfc_xqaby 2022-08-17T13:05:44.8549389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfc_xqaby/_remote_module_non_scriptable.py 2022-08-17T13:05:44.8658200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ch4j7j9 2022-08-17T13:05:44.8659163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ch4j7j9/_remote_module_non_scriptable.py 2022-08-17T13:05:45.0782569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:45.1011522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:45.1175308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:45.1249066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:45.8445391Z ok (2.480s) 2022-08-17T13:05:45.8445659Z 2022-08-17T13:05:45.8446132Z ---------------------------------------------------------------------- 2022-08-17T13:05:45.8446418Z Ran 1 test in 2.480s 2022-08-17T13:05:45.8446541Z 2022-08-17T13:05:45.8446605Z OK 2022-08-17T13:05:45.8446709Z 2022-08-17T13:05:45.8446804Z Generating XML reports... 2022-08-17T13:05:45.8480158Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130543.xml 2022-08-17T13:05:47.1683382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:47.1684055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:47.1687074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:47.1688073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:47.3193294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zlw93vs 2022-08-17T13:05:47.3196451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zlw93vs/_remote_module_non_scriptable.py 2022-08-17T13:05:47.5812455Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:47.5823098Z 2022-08-17T13:05:47.5823236Z Running tests... 2022-08-17T13:05:47.5823925Z ---------------------------------------------------------------------- 2022-08-17T13:05:47.8473653Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22965 2022-08-17T13:05:47.8503052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22966 2022-08-17T13:05:47.8531667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22967 2022-08-17T13:05:47.8561989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22968 2022-08-17T13:05:48.9078056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:48.9078755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:48.9081349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:48.9082014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:48.9154910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:48.9155567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:48.9159153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:48.9159851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:48.9569980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:48.9570541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:48.9572411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:48.9572952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:48.9656077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:48.9656631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:48.9660127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:48.9660842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:49.0688872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivwhz7pz 2022-08-17T13:05:49.0689600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivwhz7pz/_remote_module_non_scriptable.py 2022-08-17T13:05:49.0742600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0u_e_d6 2022-08-17T13:05:49.0743610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0u_e_d6/_remote_module_non_scriptable.py 2022-08-17T13:05:49.1169171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwxjw081 2022-08-17T13:05:49.1170014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwxjw081/_remote_module_non_scriptable.py 2022-08-17T13:05:49.1257425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwex3g2m 2022-08-17T13:05:49.1258669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwex3g2m/_remote_module_non_scriptable.py 2022-08-17T13:05:49.3297577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:49.3325070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:49.3780311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:49.3904983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:49.4371510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:05:49.4371945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:05:49.4472514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:05:49.4473137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:05:49.4474097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:49.4475217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:49.4476050Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:49.4476918Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:49.9314114Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:50.1624096Z ok (2.580s) 2022-08-17T13:05:50.1624332Z 2022-08-17T13:05:50.1624850Z ---------------------------------------------------------------------- 2022-08-17T13:05:50.1625357Z Ran 1 test in 2.580s 2022-08-17T13:05:50.1625572Z 2022-08-17T13:05:50.1625668Z OK 2022-08-17T13:05:50.1625839Z 2022-08-17T13:05:50.1626011Z Generating XML reports... 2022-08-17T13:05:50.1659240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130547.xml 2022-08-17T13:05:51.4966217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:51.4966601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:51.4968690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:51.4969067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:51.6515581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqaqgcsy 2022-08-17T13:05:51.6519090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqaqgcsy/_remote_module_non_scriptable.py 2022-08-17T13:05:51.9205262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:51.9215630Z 2022-08-17T13:05:51.9215778Z Running tests... 2022-08-17T13:05:51.9216336Z ---------------------------------------------------------------------- 2022-08-17T13:05:52.1900914Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23175 2022-08-17T13:05:52.1929226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23176 2022-08-17T13:05:52.1958636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23177 2022-08-17T13:05:52.1990768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23178 2022-08-17T13:05:53.1919498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:53.1920124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:53.1923160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:53.1923824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:53.2057447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:53.2058092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:53.2060774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:53.2061447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:53.2305431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:53.2306071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:53.2308850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:53.2309776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:53.2312962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:53.2313614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:53.2318725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:53.2319520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:53.3537478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40bc_1yh 2022-08-17T13:05:53.3538772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40bc_1yh/_remote_module_non_scriptable.py 2022-08-17T13:05:53.3672908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzjbtqv2 2022-08-17T13:05:53.3673893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzjbtqv2/_remote_module_non_scriptable.py 2022-08-17T13:05:53.3915499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_27r1aj 2022-08-17T13:05:53.3916336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_27r1aj/_remote_module_non_scriptable.py 2022-08-17T13:05:53.3935783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz2yl349 2022-08-17T13:05:53.3938045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz2yl349/_remote_module_non_scriptable.py 2022-08-17T13:05:53.6206548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:53.6353182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:53.6560656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:53.6596879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:53.6876704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:05:53.6977337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:05:53.7080939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:05:53.7081537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:05:53.7082302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:53.7083077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:53.7083627Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:53.7084161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:05:54.1904856Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.1906229Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.1907499Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.2055480Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.2057124Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.2094316Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:05:54.6056885Z ok (2.684s) 2022-08-17T13:05:54.6057079Z 2022-08-17T13:05:54.6057412Z ---------------------------------------------------------------------- 2022-08-17T13:05:54.6057722Z Ran 1 test in 2.684s 2022-08-17T13:05:54.6057844Z 2022-08-17T13:05:54.6057925Z OK 2022-08-17T13:05:54.6058021Z 2022-08-17T13:05:54.6058115Z Generating XML reports... 2022-08-17T13:05:54.6091073Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130551.xml 2022-08-17T13:05:55.9213548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:55.9213962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:55.9216386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:55.9216768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:56.0744821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_50orzhx 2022-08-17T13:05:56.0747763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_50orzhx/_remote_module_non_scriptable.py 2022-08-17T13:05:56.3384796Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:05:56.3395401Z 2022-08-17T13:05:56.3395509Z Running tests... 2022-08-17T13:05:56.3396055Z ---------------------------------------------------------------------- 2022-08-17T13:05:56.6035063Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23398 2022-08-17T13:05:56.6063886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23399 2022-08-17T13:05:56.6092394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23400 2022-08-17T13:05:56.6124702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23401 2022-08-17T13:05:57.6351117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:57.6351749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:57.6354758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:57.6355374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:57.6380604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:57.6383933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:57.6384780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:57.6385416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:57.6399939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:57.6400593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:57.6404590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:57.6405246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:57.6551247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:57.6551924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:57.6554528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:57.6555181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:57.7955690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4wgyama 2022-08-17T13:05:57.7956948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4wgyama/_remote_module_non_scriptable.py 2022-08-17T13:05:57.7963241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyq2o8_9 2022-08-17T13:05:57.7966244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyq2o8_9/_remote_module_non_scriptable.py 2022-08-17T13:05:57.7989306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtfqdh1_ 2022-08-17T13:05:57.7991908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtfqdh1_/_remote_module_non_scriptable.py 2022-08-17T13:05:57.8127818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe_rh99l 2022-08-17T13:05:57.8129053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe_rh99l/_remote_module_non_scriptable.py 2022-08-17T13:05:58.0559766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:05:58.0571421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:05:58.0587638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:05:58.0770310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:05:58.4176968Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T13:05:58.4177211Z 2022-08-17T13:05:58.4177781Z ---------------------------------------------------------------------- 2022-08-17T13:05:58.4178164Z Ran 1 test in 2.078s 2022-08-17T13:05:58.4178286Z 2022-08-17T13:05:58.4178365Z OK (skipped=1) 2022-08-17T13:05:58.4178480Z 2022-08-17T13:05:58.4178556Z Generating XML reports... 2022-08-17T13:05:58.4212671Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130556.xml 2022-08-17T13:05:59.7223188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:05:59.7223573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:05:59.7226636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:05:59.7227137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:05:59.8765442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor33uz09 2022-08-17T13:05:59.8769580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor33uz09/_remote_module_non_scriptable.py 2022-08-17T13:06:00.1425486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:00.1436070Z 2022-08-17T13:06:00.1436187Z Running tests... 2022-08-17T13:06:00.1436823Z ---------------------------------------------------------------------- 2022-08-17T13:06:00.4103659Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23453 2022-08-17T13:06:00.4132122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23454 2022-08-17T13:06:00.4162274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23455 2022-08-17T13:06:00.4196103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23456 2022-08-17T13:06:01.4355845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:01.4356533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:01.4359577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:01.4360245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:01.4382467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:01.4383131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:01.4386733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:01.4387398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:01.4759373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:01.4760106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:01.4762250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:01.4762928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:01.4763751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:01.4764459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:01.4768205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:01.4769053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:01.5985127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fw_8yzl 2022-08-17T13:06:01.5986403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavstxnvw 2022-08-17T13:06:01.5987365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fw_8yzl/_remote_module_non_scriptable.py 2022-08-17T13:06:01.5988384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavstxnvw/_remote_module_non_scriptable.py 2022-08-17T13:06:01.6378955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu3lel_h 2022-08-17T13:06:01.6380060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn77ktggq 2022-08-17T13:06:01.6380878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu3lel_h/_remote_module_non_scriptable.py 2022-08-17T13:06:01.6381894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn77ktggq/_remote_module_non_scriptable.py 2022-08-17T13:06:01.8559380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:01.8711097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:01.9002246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:01.9124301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:02.5254154Z ok (2.381s) 2022-08-17T13:06:02.5254374Z 2022-08-17T13:06:02.5254907Z ---------------------------------------------------------------------- 2022-08-17T13:06:02.5255378Z Ran 1 test in 2.382s 2022-08-17T13:06:02.5255588Z 2022-08-17T13:06:02.5255689Z OK 2022-08-17T13:06:02.5255789Z 2022-08-17T13:06:02.5255887Z Generating XML reports... 2022-08-17T13:06:02.5288796Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130600.xml 2022-08-17T13:06:03.8087837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:03.8088521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:03.8090536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:03.8090919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:03.9634006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yay_m7r 2022-08-17T13:06:03.9637802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yay_m7r/_remote_module_non_scriptable.py 2022-08-17T13:06:04.2284762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:04.2295741Z 2022-08-17T13:06:04.2295884Z Running tests... 2022-08-17T13:06:04.2296395Z ---------------------------------------------------------------------- 2022-08-17T13:06:04.5019781Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23664 2022-08-17T13:06:04.5050270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23665 2022-08-17T13:06:04.5080398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23666 2022-08-17T13:06:04.5114472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23667 2022-08-17T13:06:05.5552786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:05.5553439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:05.5556551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:05.5557270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:05.5812927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:05.5813618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:05.5816732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:05.5817328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:05.6029363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:05.6030014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:05.6033267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:05.6034168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:05.6141987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:05.6142527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:05.6146058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:05.6146747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:05.7141223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcpaco5t 2022-08-17T13:06:05.7141970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcpaco5t/_remote_module_non_scriptable.py 2022-08-17T13:06:05.7413756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswpocnf3 2022-08-17T13:06:05.7414829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswpocnf3/_remote_module_non_scriptable.py 2022-08-17T13:06:05.7627787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zffnppd 2022-08-17T13:06:05.7628641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zffnppd/_remote_module_non_scriptable.py 2022-08-17T13:06:05.7748113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6fjc9tm 2022-08-17T13:06:05.7748999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6fjc9tm/_remote_module_non_scriptable.py 2022-08-17T13:06:05.9746779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:06.0056807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:06.0273308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:06.0380678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:06.4171462Z ok (2.187s) 2022-08-17T13:06:06.4171636Z 2022-08-17T13:06:06.4172512Z ---------------------------------------------------------------------- 2022-08-17T13:06:06.4172964Z Ran 1 test in 2.187s 2022-08-17T13:06:06.4173167Z 2022-08-17T13:06:06.4173265Z OK 2022-08-17T13:06:06.4173427Z 2022-08-17T13:06:06.4173566Z Generating XML reports... 2022-08-17T13:06:06.4236782Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130604.xml 2022-08-17T13:06:07.7274029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:07.7274633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:07.7276892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:07.7277462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:07.8802898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69e36t85 2022-08-17T13:06:07.8806741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69e36t85/_remote_module_non_scriptable.py 2022-08-17T13:06:08.1446812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:08.1457524Z 2022-08-17T13:06:08.1457813Z Running tests... 2022-08-17T13:06:08.1458487Z ---------------------------------------------------------------------- 2022-08-17T13:06:08.4060970Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23723 2022-08-17T13:06:08.4089629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23724 2022-08-17T13:06:08.4117912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23725 2022-08-17T13:06:08.4149809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23726 2022-08-17T13:06:09.3881557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:09.3884997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:09.3922718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:09.3923389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:09.4308658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:09.4309319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:09.4311839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:09.4312485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:09.4381759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:09.4382448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:09.4386556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:09.4387222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:09.4520665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:09.4521323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:09.4524701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:09.4525358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:09.5494903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y3g8z6w 2022-08-17T13:06:09.5496334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y3g8z6w/_remote_module_non_scriptable.py 2022-08-17T13:06:09.5985949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqvuye71 2022-08-17T13:06:09.5986807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqvuye71/_remote_module_non_scriptable.py 2022-08-17T13:06:09.6139481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmyabztc 2022-08-17T13:06:09.6140300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmyabztc/_remote_module_non_scriptable.py 2022-08-17T13:06:09.6167983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ziw_ltw 2022-08-17T13:06:09.6169742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ziw_ltw/_remote_module_non_scriptable.py 2022-08-17T13:06:09.8126953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:09.8629198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:09.8781368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:09.8803075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:09.9096064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:06:09.9198981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:06:09.9199706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:06:09.9200299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:06:09.9201247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:06:09.9202021Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:06:09.9202585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:06:09.9203124Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:06:10.3690062Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.3691417Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.3692720Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.3848035Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.3848761Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.3947134Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:10.8213815Z ok (2.675s) 2022-08-17T13:06:10.8214066Z 2022-08-17T13:06:10.8214545Z ---------------------------------------------------------------------- 2022-08-17T13:06:10.8214969Z Ran 1 test in 2.676s 2022-08-17T13:06:10.8215159Z 2022-08-17T13:06:10.8215254Z OK 2022-08-17T13:06:10.8215402Z 2022-08-17T13:06:10.8215553Z Generating XML reports... 2022-08-17T13:06:10.8249554Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130608.xml 2022-08-17T13:06:12.1107212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:12.1107691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:12.1110311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:12.1110749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:12.2615899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oi_bz1v 2022-08-17T13:06:12.2619861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oi_bz1v/_remote_module_non_scriptable.py 2022-08-17T13:06:12.5297732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:12.5308831Z 2022-08-17T13:06:12.5309328Z Running tests... 2022-08-17T13:06:12.5309768Z ---------------------------------------------------------------------- 2022-08-17T13:06:12.7989045Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23946 2022-08-17T13:06:12.8020278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23947 2022-08-17T13:06:12.8050429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23948 2022-08-17T13:06:12.8083803Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23949 2022-08-17T13:06:13.8164480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:13.8165460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:13.8167898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:13.8168489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:13.8459689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:13.8460388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:13.8463217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:13.8463951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:13.8472425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:13.8473100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:13.8477358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:13.8478289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:13.8643027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:13.8643713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:13.8647319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:13.8647965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:13.9731171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxzotxyc 2022-08-17T13:06:13.9732510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxzotxyc/_remote_module_non_scriptable.py 2022-08-17T13:06:14.0059733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquqlo30y 2022-08-17T13:06:14.0060799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquqlo30y/_remote_module_non_scriptable.py 2022-08-17T13:06:14.0082654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhstsd96 2022-08-17T13:06:14.0084385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhstsd96/_remote_module_non_scriptable.py 2022-08-17T13:06:14.0236244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt5pbnjg 2022-08-17T13:06:14.0237286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt5pbnjg/_remote_module_non_scriptable.py 2022-08-17T13:06:14.2329856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:14.2669076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:14.2713092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:14.2849853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:14.8770789Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:14.8772276Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:14.8773306Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:06:15.0145579Z ok (2.483s) 2022-08-17T13:06:15.0145806Z 2022-08-17T13:06:15.0146581Z ---------------------------------------------------------------------- 2022-08-17T13:06:15.0146894Z Ran 1 test in 2.484s 2022-08-17T13:06:15.0147002Z 2022-08-17T13:06:15.0147069Z OK 2022-08-17T13:06:15.0147209Z 2022-08-17T13:06:15.0147380Z Generating XML reports... 2022-08-17T13:06:15.0180369Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130612.xml 2022-08-17T13:06:16.3096893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:16.3097280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:16.3099796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:16.3100213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:16.4592482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppou0h1te 2022-08-17T13:06:16.4595210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppou0h1te/_remote_module_non_scriptable.py 2022-08-17T13:06:16.7209603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:16.7220358Z 2022-08-17T13:06:16.7220779Z Running tests... 2022-08-17T13:06:16.7221414Z ---------------------------------------------------------------------- 2022-08-17T13:06:16.9827786Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24169 2022-08-17T13:06:16.9856404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24170 2022-08-17T13:06:16.9885539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24171 2022-08-17T13:06:16.9917353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24172 2022-08-17T13:06:18.0006608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:18.0007263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:18.0009433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:18.0010010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:18.0240155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:18.0240804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:18.0243675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:18.0244338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:18.0254433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:18.0255079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:18.0259090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:18.0259761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:18.0376618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:18.0377314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:18.0380890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:18.0381536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:18.1601052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrxwihyd 2022-08-17T13:06:18.1602182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrxwihyd/_remote_module_non_scriptable.py 2022-08-17T13:06:18.1823363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyjg2ee7 2022-08-17T13:06:18.1824252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyjg2ee7/_remote_module_non_scriptable.py 2022-08-17T13:06:18.1840626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvwvrn88 2022-08-17T13:06:18.1842588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvwvrn88/_remote_module_non_scriptable.py 2022-08-17T13:06:18.1977362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptos1alb3 2022-08-17T13:06:18.1978798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptos1alb3/_remote_module_non_scriptable.py 2022-08-17T13:06:18.4218921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:18.4416878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:18.4434077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:18.4611926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:18.7971026Z ok (2.075s) 2022-08-17T13:06:18.7971327Z 2022-08-17T13:06:18.7971683Z ---------------------------------------------------------------------- 2022-08-17T13:06:18.7971964Z Ran 1 test in 2.075s 2022-08-17T13:06:18.7972087Z 2022-08-17T13:06:18.7972157Z OK 2022-08-17T13:06:18.7972241Z 2022-08-17T13:06:18.7972338Z Generating XML reports... 2022-08-17T13:06:18.8007821Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130616.xml 2022-08-17T13:06:20.1079517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:20.1080217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:20.1083356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:20.1083975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:20.2596950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9sxig_r 2022-08-17T13:06:20.2600738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9sxig_r/_remote_module_non_scriptable.py 2022-08-17T13:06:20.5252589Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:20.5264061Z 2022-08-17T13:06:20.5264592Z Running tests... 2022-08-17T13:06:20.5265211Z ---------------------------------------------------------------------- 2022-08-17T13:06:20.7915115Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24224 2022-08-17T13:06:20.7944504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24225 2022-08-17T13:06:20.7973092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24226 2022-08-17T13:06:20.8005519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24227 2022-08-17T13:06:21.8241093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:21.8241734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:21.8244281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:21.8244808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:21.8278246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:21.8279268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:21.8283478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:21.8284118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:21.8497227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:21.8497913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:21.8500239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:21.8500883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:21.8603198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:21.8603872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:21.8607237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:21.8607922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:21.9855968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoizrpjj6 2022-08-17T13:06:21.9857589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoizrpjj6/_remote_module_non_scriptable.py 2022-08-17T13:06:21.9901304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dqpjywj 2022-08-17T13:06:21.9903434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dqpjywj/_remote_module_non_scriptable.py 2022-08-17T13:06:22.0100486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputkan00m 2022-08-17T13:06:22.0101787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputkan00m/_remote_module_non_scriptable.py 2022-08-17T13:06:22.0198846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv2xcrnw 2022-08-17T13:06:22.0200141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv2xcrnw/_remote_module_non_scriptable.py 2022-08-17T13:06:22.2622715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:22.2729270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:22.2747923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:22.2860466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:23.0065642Z ok (2.480s) 2022-08-17T13:06:23.0065895Z 2022-08-17T13:06:23.0066438Z ---------------------------------------------------------------------- 2022-08-17T13:06:23.0066765Z Ran 1 test in 2.480s 2022-08-17T13:06:23.0066887Z 2022-08-17T13:06:23.0066963Z OK 2022-08-17T13:06:23.0067060Z 2022-08-17T13:06:23.0067156Z Generating XML reports... 2022-08-17T13:06:23.0099561Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130620.xml 2022-08-17T13:06:24.3169000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:24.3169635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:24.3172121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:24.3172729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:24.4663779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eg8ly54 2022-08-17T13:06:24.4666017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eg8ly54/_remote_module_non_scriptable.py 2022-08-17T13:06:24.7290783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:24.7301286Z 2022-08-17T13:06:24.7301386Z Running tests... 2022-08-17T13:06:24.7301851Z ---------------------------------------------------------------------- 2022-08-17T13:06:24.9901262Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24459 2022-08-17T13:06:24.9929387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24460 2022-08-17T13:06:24.9958280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24461 2022-08-17T13:06:24.9989769Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24462 2022-08-17T13:06:26.0519994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:26.0520613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:26.0524019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:26.0524661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:26.0526217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:26.0526793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:26.0531513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:26.0532209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:26.0639046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:26.0639674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:26.0642642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:26.0643284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:26.0681224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:26.0681900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:26.0686245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:26.0686890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:26.2131415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcxl6hup 2022-08-17T13:06:26.2132289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcxl6hup/_remote_module_non_scriptable.py 2022-08-17T13:06:26.2134782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02eo9e4v 2022-08-17T13:06:26.2137822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02eo9e4v/_remote_module_non_scriptable.py 2022-08-17T13:06:26.2275211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_4v8daj 2022-08-17T13:06:26.2276622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_4v8daj/_remote_module_non_scriptable.py 2022-08-17T13:06:26.2397921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnzz17ll 2022-08-17T13:06:26.2399163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnzz17ll/_remote_module_non_scriptable.py 2022-08-17T13:06:26.4770672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:26.4772667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:26.4851092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:26.5106104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:27.2050504Z ok (2.475s) 2022-08-17T13:06:27.2050683Z 2022-08-17T13:06:27.2051201Z ---------------------------------------------------------------------- 2022-08-17T13:06:27.2051659Z Ran 1 test in 2.475s 2022-08-17T13:06:27.2051781Z 2022-08-17T13:06:27.2051846Z OK 2022-08-17T13:06:27.2051941Z 2022-08-17T13:06:27.2052037Z Generating XML reports... 2022-08-17T13:06:27.2087974Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130624.xml 2022-08-17T13:06:28.5159826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:28.5160480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:28.5161841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:28.5162468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:28.6669754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesg2g3lb 2022-08-17T13:06:28.6673371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesg2g3lb/_remote_module_non_scriptable.py 2022-08-17T13:06:28.9282113Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:28.9293549Z 2022-08-17T13:06:28.9293965Z Running tests... 2022-08-17T13:06:28.9294377Z ---------------------------------------------------------------------- 2022-08-17T13:06:29.1932556Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24694 2022-08-17T13:06:29.1961557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24695 2022-08-17T13:06:29.1990532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24696 2022-08-17T13:06:29.2022786Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24697 2022-08-17T13:06:30.2289690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:30.2290394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:30.2292829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:30.2293450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:30.2487697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:30.2488349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:30.2489890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:30.2490497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:30.2548154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:30.2548873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:30.2552976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:30.2553624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:30.2580324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:30.2580992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:30.2584589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:30.2585550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:30.3899190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu58pk6g 2022-08-17T13:06:30.3900407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu58pk6g/_remote_module_non_scriptable.py 2022-08-17T13:06:30.4181886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeji56191 2022-08-17T13:06:30.4183081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeji56191/_remote_module_non_scriptable.py 2022-08-17T13:06:30.4193081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnwrkd41 2022-08-17T13:06:30.4195968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnwrkd41/_remote_module_non_scriptable.py 2022-08-17T13:06:30.4338627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp1sgzy5 2022-08-17T13:06:30.4339817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp1sgzy5/_remote_module_non_scriptable.py 2022-08-17T13:06:30.6610670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:30.6809189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:30.6846326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:30.6987227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:32.7112782Z ok (3.782s) 2022-08-17T13:06:32.7114037Z 2022-08-17T13:06:32.7114596Z ---------------------------------------------------------------------- 2022-08-17T13:06:32.7115034Z Ran 1 test in 3.782s 2022-08-17T13:06:32.7115231Z 2022-08-17T13:06:32.7115346Z OK 2022-08-17T13:06:32.7115515Z 2022-08-17T13:06:32.7115684Z Generating XML reports... 2022-08-17T13:06:32.7150988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130628.xml 2022-08-17T13:06:34.0208787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:34.0209169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:34.0211545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:34.0211925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:34.1715311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xgt7bpf 2022-08-17T13:06:34.1718845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xgt7bpf/_remote_module_non_scriptable.py 2022-08-17T13:06:34.4369769Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:34.4380414Z 2022-08-17T13:06:34.4380513Z Running tests... 2022-08-17T13:06:34.4381002Z ---------------------------------------------------------------------- 2022-08-17T13:06:34.7090343Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24929 2022-08-17T13:06:34.7119773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24930 2022-08-17T13:06:34.7148878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24931 2022-08-17T13:06:34.7180709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24932 2022-08-17T13:06:35.7801761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:35.7802573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:35.7803332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:35.7804259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:35.7805071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:35.7805685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:35.7806721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:35.7807324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:35.7833243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:35.7833886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:35.7837323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:35.7838006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:35.7986035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:35.7986741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:35.7990525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:35.7991202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:35.9421522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21l8vml1 2022-08-17T13:06:35.9422288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21l8vml1/_remote_module_non_scriptable.py 2022-08-17T13:06:35.9426072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgca_4bdq 2022-08-17T13:06:35.9427066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjc5s2qh 2022-08-17T13:06:35.9428063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgca_4bdq/_remote_module_non_scriptable.py 2022-08-17T13:06:35.9430450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjc5s2qh/_remote_module_non_scriptable.py 2022-08-17T13:06:35.9598536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76_t2ai8 2022-08-17T13:06:35.9599592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76_t2ai8/_remote_module_non_scriptable.py 2022-08-17T13:06:36.2052844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:36.2057097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:36.2079744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:36.2276712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:37.0243972Z ok (2.586s) 2022-08-17T13:06:37.0244174Z 2022-08-17T13:06:37.0244650Z ---------------------------------------------------------------------- 2022-08-17T13:06:37.0245120Z Ran 1 test in 2.586s 2022-08-17T13:06:37.0245243Z 2022-08-17T13:06:37.0245308Z OK 2022-08-17T13:06:37.0245403Z 2022-08-17T13:06:37.0245502Z Generating XML reports... 2022-08-17T13:06:37.0281543Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130634.xml 2022-08-17T13:06:38.3371586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:38.3372295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:38.3375245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:38.3375819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:38.4908979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnuep6d1 2022-08-17T13:06:38.4912752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnuep6d1/_remote_module_non_scriptable.py 2022-08-17T13:06:38.7567523Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:38.7578955Z 2022-08-17T13:06:38.7579487Z Running tests... 2022-08-17T13:06:38.7579996Z ---------------------------------------------------------------------- 2022-08-17T13:06:39.0279989Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25188 2022-08-17T13:06:39.0309294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25189 2022-08-17T13:06:39.0339961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25190 2022-08-17T13:06:39.0373368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25191 2022-08-17T13:06:40.1146930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:40.1147574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:40.1150647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:40.1151318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:40.1187974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:40.1188622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:40.1192142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:40.1192806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:40.1226803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:40.1227449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:40.1231197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:40.1231823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:40.1315385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:40.1316058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:40.1319728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:40.1320335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:40.2750785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqp1688l 2022-08-17T13:06:40.2751864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqp1688l/_remote_module_non_scriptable.py 2022-08-17T13:06:40.2793621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuz0b0by 2022-08-17T13:06:40.2795644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuz0b0by/_remote_module_non_scriptable.py 2022-08-17T13:06:40.2822479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1elokn5e 2022-08-17T13:06:40.2824948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1elokn5e/_remote_module_non_scriptable.py 2022-08-17T13:06:40.2922543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5sg0yn1 2022-08-17T13:06:40.2923811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5sg0yn1/_remote_module_non_scriptable.py 2022-08-17T13:06:40.5371597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:40.5403747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:40.5423507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:40.5575962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:41.3434228Z ok (2.585s) 2022-08-17T13:06:41.3434463Z 2022-08-17T13:06:41.3434816Z ---------------------------------------------------------------------- 2022-08-17T13:06:41.3435150Z Ran 1 test in 2.585s 2022-08-17T13:06:41.3435272Z 2022-08-17T13:06:41.3435340Z OK 2022-08-17T13:06:41.3435434Z 2022-08-17T13:06:41.3435533Z Generating XML reports... 2022-08-17T13:06:41.3469405Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130638.xml 2022-08-17T13:06:42.6425614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:42.6426012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:42.6428326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:42.6429033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:42.7956054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1zngw9m 2022-08-17T13:06:42.7959909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1zngw9m/_remote_module_non_scriptable.py 2022-08-17T13:06:43.0604586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:43.0615420Z 2022-08-17T13:06:43.0615553Z Running tests... 2022-08-17T13:06:43.0616168Z ---------------------------------------------------------------------- 2022-08-17T13:06:43.3211162Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25435 2022-08-17T13:06:43.3238734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25436 2022-08-17T13:06:43.3267130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25437 2022-08-17T13:06:43.3298494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25438 2022-08-17T13:06:44.3170473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:44.3171057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:44.3173741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:44.3174316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:44.3530182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:44.3530818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:44.3533648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:44.3534287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:44.3648746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:44.3649366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:44.3652814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:44.3653393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:44.3722164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:44.3723104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:44.3726079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:44.3726655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:44.4743378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2ztcds0 2022-08-17T13:06:44.4744288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2ztcds0/_remote_module_non_scriptable.py 2022-08-17T13:06:44.5132669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2p05qz7 2022-08-17T13:06:44.5133529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2p05qz7/_remote_module_non_scriptable.py 2022-08-17T13:06:44.5239342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmowl5xxh 2022-08-17T13:06:44.5240417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmowl5xxh/_remote_module_non_scriptable.py 2022-08-17T13:06:44.5310802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5co4dnw4 2022-08-17T13:06:44.5312130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5co4dnw4/_remote_module_non_scriptable.py 2022-08-17T13:06:44.7336944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:44.7749244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:44.7840216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:44.7920005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:45.5358024Z ok (2.474s) 2022-08-17T13:06:45.5358305Z 2022-08-17T13:06:45.5358816Z ---------------------------------------------------------------------- 2022-08-17T13:06:45.5359069Z Ran 1 test in 2.474s 2022-08-17T13:06:45.5359204Z 2022-08-17T13:06:45.5359268Z OK 2022-08-17T13:06:45.5359364Z 2022-08-17T13:06:45.5359458Z Generating XML reports... 2022-08-17T13:06:45.5393448Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130643.xml 2022-08-17T13:06:46.8115773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:46.8116388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:46.8117979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:46.8118598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:46.9652858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5onf6kpr 2022-08-17T13:06:46.9656764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5onf6kpr/_remote_module_non_scriptable.py 2022-08-17T13:06:47.2348888Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:47.2359555Z 2022-08-17T13:06:47.2359649Z Running tests... 2022-08-17T13:06:47.2360232Z ---------------------------------------------------------------------- 2022-08-17T13:06:47.5016175Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25682 2022-08-17T13:06:47.5044821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25683 2022-08-17T13:06:47.5073978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25684 2022-08-17T13:06:47.5106313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25685 2022-08-17T13:06:48.6083136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:48.6083839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:48.6087147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:48.6088109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:48.6181595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:48.6182327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:48.6185974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:48.6186601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:48.6408840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:48.6409500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:48.6412827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:48.6413497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:48.6545080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:48.6545781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:48.6548691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:48.6549353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:48.7690947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgu5rjoa 2022-08-17T13:06:48.7703774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgu5rjoa/_remote_module_non_scriptable.py 2022-08-17T13:06:48.7789653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oys5ag6 2022-08-17T13:06:48.7791006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oys5ag6/_remote_module_non_scriptable.py 2022-08-17T13:06:48.8007349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv86oqa6 2022-08-17T13:06:48.8008225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv86oqa6/_remote_module_non_scriptable.py 2022-08-17T13:06:48.8174048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1icgu21o 2022-08-17T13:06:48.8175039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1icgu21o/_remote_module_non_scriptable.py 2022-08-17T13:06:49.0339310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:49.0445275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:49.0625010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:49.0827205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:49.9168714Z ok (2.681s) 2022-08-17T13:06:49.9168950Z 2022-08-17T13:06:49.9169291Z ---------------------------------------------------------------------- 2022-08-17T13:06:49.9169559Z Ran 1 test in 2.681s 2022-08-17T13:06:49.9169681Z 2022-08-17T13:06:49.9169745Z OK 2022-08-17T13:06:49.9169840Z 2022-08-17T13:06:49.9169936Z Generating XML reports... 2022-08-17T13:06:49.9205207Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130647.xml 2022-08-17T13:06:51.2017093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:51.2017486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:51.2020518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:51.2021181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:51.3511043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_fbqm7o 2022-08-17T13:06:51.3513779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_fbqm7o/_remote_module_non_scriptable.py 2022-08-17T13:06:51.6128856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:51.6139122Z 2022-08-17T13:06:51.6139212Z Running tests... 2022-08-17T13:06:51.6139970Z ---------------------------------------------------------------------- 2022-08-17T13:06:51.8731957Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25929 2022-08-17T13:06:51.8759339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25930 2022-08-17T13:06:51.8787380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25931 2022-08-17T13:06:51.8819651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25932 2022-08-17T13:06:52.9445032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:52.9445727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:52.9448135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:52.9448703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:52.9818078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:52.9818797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:52.9821469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:52.9822138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:53.0173034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:53.0173796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:53.0176527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:53.0177326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:53.0200829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:53.0201574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:53.0205377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:53.0206358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:53.1030507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbl0p8ni 2022-08-17T13:06:53.1031713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbl0p8ni/_remote_module_non_scriptable.py 2022-08-17T13:06:53.1387590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsm63lb7 2022-08-17T13:06:53.1388469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsm63lb7/_remote_module_non_scriptable.py 2022-08-17T13:06:53.1754626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g5vf3d9 2022-08-17T13:06:53.1756156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g5vf3d9/_remote_module_non_scriptable.py 2022-08-17T13:06:53.1777810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmfj8h2w 2022-08-17T13:06:53.1779231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmfj8h2w/_remote_module_non_scriptable.py 2022-08-17T13:06:53.3630716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:53.3982062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:53.4343052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:53.4381304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:54.3885776Z ok (2.774s) 2022-08-17T13:06:54.3886200Z 2022-08-17T13:06:54.3886735Z ---------------------------------------------------------------------- 2022-08-17T13:06:54.3887119Z Ran 1 test in 2.775s 2022-08-17T13:06:54.3887244Z 2022-08-17T13:06:54.3887295Z OK 2022-08-17T13:06:54.3887390Z 2022-08-17T13:06:54.3887485Z Generating XML reports... 2022-08-17T13:06:54.3920377Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130651.xml 2022-08-17T13:06:55.6901346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:55.6901767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:55.6904394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:55.6904819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:55.8420811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3b0hzw4 2022-08-17T13:06:55.8424200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3b0hzw4/_remote_module_non_scriptable.py 2022-08-17T13:06:56.1066573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:06:56.1077825Z 2022-08-17T13:06:56.1078116Z Running tests... 2022-08-17T13:06:56.1078557Z ---------------------------------------------------------------------- 2022-08-17T13:06:56.3722871Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26356 2022-08-17T13:06:56.3751824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26357 2022-08-17T13:06:56.3782357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26358 2022-08-17T13:06:56.3814868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26359 2022-08-17T13:06:57.4088873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:57.4089587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:57.4092391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:57.4093097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:57.4202898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:57.4203544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:57.4207076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:57.4207703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:57.4222514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:57.4223096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:57.4227771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:57.4228383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:57.4229491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:57.4230051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:57.4232588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:57.4233249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:57.5693418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80omqxlr 2022-08-17T13:06:57.5694212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80omqxlr/_remote_module_non_scriptable.py 2022-08-17T13:06:57.5791546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps92i694o 2022-08-17T13:06:57.5792951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps92i694o/_remote_module_non_scriptable.py 2022-08-17T13:06:57.5830704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehzik7ep 2022-08-17T13:06:57.5833410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehzik7ep/_remote_module_non_scriptable.py 2022-08-17T13:06:57.5837005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4_afova 2022-08-17T13:06:57.5839314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4_afova/_remote_module_non_scriptable.py 2022-08-17T13:06:57.8321518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:06:57.8426582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:06:57.8491048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:06:57.8702887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:06:58.4872913Z ok (2.379s) 2022-08-17T13:06:58.4873138Z 2022-08-17T13:06:58.4873590Z ---------------------------------------------------------------------- 2022-08-17T13:06:58.4874009Z Ran 1 test in 2.379s 2022-08-17T13:06:58.4874193Z 2022-08-17T13:06:58.4874305Z OK 2022-08-17T13:06:58.4874464Z 2022-08-17T13:06:58.4874614Z Generating XML reports... 2022-08-17T13:06:58.4909501Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130656.xml 2022-08-17T13:06:59.8052855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:06:59.8053260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:06:59.8055674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:06:59.8056178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:06:59.9571679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8ez3ck8 2022-08-17T13:06:59.9574630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8ez3ck8/_remote_module_non_scriptable.py 2022-08-17T13:07:00.2228784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:00.2238825Z 2022-08-17T13:07:00.2238925Z Running tests... 2022-08-17T13:07:00.2239342Z ---------------------------------------------------------------------- 2022-08-17T13:07:00.4894947Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26579 2022-08-17T13:07:00.4922858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26580 2022-08-17T13:07:00.4951481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26581 2022-08-17T13:07:00.4983049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26582 2022-08-17T13:07:01.5323387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:01.5324049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:01.5327915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:01.5328616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:01.5633590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:01.5634269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:01.5635391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:01.5636038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:01.5961371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:01.5962028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:01.5963339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:01.5964011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:01.6353414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:01.6354038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:01.6357140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:01.6357811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:01.6958949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2e8rcr3 2022-08-17T13:07:01.6960100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2e8rcr3/_remote_module_non_scriptable.py 2022-08-17T13:07:01.7258297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9qzpdkx 2022-08-17T13:07:01.7259305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9qzpdkx/_remote_module_non_scriptable.py 2022-08-17T13:07:01.7575048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01veb87_ 2022-08-17T13:07:01.7575758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01veb87_/_remote_module_non_scriptable.py 2022-08-17T13:07:01.7964975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflff_m6k 2022-08-17T13:07:01.7965809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflff_m6k/_remote_module_non_scriptable.py 2022-08-17T13:07:01.9645683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:02.0135901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:02.0228273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:02.0627814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:03.2053723Z ok (2.981s) 2022-08-17T13:07:03.2053963Z 2022-08-17T13:07:03.2054434Z ---------------------------------------------------------------------- 2022-08-17T13:07:03.2054833Z Ran 1 test in 2.981s 2022-08-17T13:07:03.2055010Z 2022-08-17T13:07:03.2055111Z OK 2022-08-17T13:07:03.2056714Z 2022-08-17T13:07:03.2057016Z Generating XML reports... 2022-08-17T13:07:03.2090919Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130700.xml 2022-08-17T13:07:04.5195182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:04.5195834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:04.5197344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:04.5197904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:04.6719269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgh3yazo 2022-08-17T13:07:04.6722758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgh3yazo/_remote_module_non_scriptable.py 2022-08-17T13:07:04.9395176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:04.9415264Z 2022-08-17T13:07:04.9415480Z Running tests... 2022-08-17T13:07:04.9415883Z ---------------------------------------------------------------------- 2022-08-17T13:07:05.2133924Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26814 2022-08-17T13:07:05.2165278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26815 2022-08-17T13:07:05.2195498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26816 2022-08-17T13:07:05.2227590Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26817 2022-08-17T13:07:06.2956320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:06.2956956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:06.2959390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:06.2960060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:06.3254656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:06.3255314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:06.3257856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:06.3258511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:06.3556662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:06.3557305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:06.3560173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:06.3560852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:06.3622182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:06.3622761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:06.3626571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:06.3627148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:06.4571582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmfhvzw7 2022-08-17T13:07:06.4572323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmfhvzw7/_remote_module_non_scriptable.py 2022-08-17T13:07:06.4845289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6rkebqv 2022-08-17T13:07:06.4846627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6rkebqv/_remote_module_non_scriptable.py 2022-08-17T13:07:06.5148014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2aorh8b 2022-08-17T13:07:06.5148965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2aorh8b/_remote_module_non_scriptable.py 2022-08-17T13:07:06.5203179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1v1wz8s 2022-08-17T13:07:06.5204691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1v1wz8s/_remote_module_non_scriptable.py 2022-08-17T13:07:06.7195956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:06.7452915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:06.7766979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:06.7823254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:07.7293077Z ok (2.787s) 2022-08-17T13:07:07.7293309Z 2022-08-17T13:07:07.7293791Z ---------------------------------------------------------------------- 2022-08-17T13:07:07.7294241Z Ran 1 test in 2.788s 2022-08-17T13:07:07.7294439Z 2022-08-17T13:07:07.7294537Z OK 2022-08-17T13:07:07.7294687Z 2022-08-17T13:07:07.7294832Z Generating XML reports... 2022-08-17T13:07:07.7329333Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130704.xml 2022-08-17T13:07:09.0516230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:09.0516637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:09.0519327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:09.0519753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:09.2037127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s6990g7 2022-08-17T13:07:09.2040274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s6990g7/_remote_module_non_scriptable.py 2022-08-17T13:07:09.4674208Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:09.4684453Z 2022-08-17T13:07:09.4684782Z Running tests... 2022-08-17T13:07:09.4685473Z ---------------------------------------------------------------------- 2022-08-17T13:07:09.7305265Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27049 2022-08-17T13:07:09.7334547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27050 2022-08-17T13:07:09.7365518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27051 2022-08-17T13:07:09.7398900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27052 2022-08-17T13:07:10.7967502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:10.7968140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:10.7970132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:10.7970759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:10.7987273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:10.7988045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:10.7992274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:10.7992910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:10.8166655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:10.8167310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:10.8170372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:10.8171052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:10.8285463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:10.8286394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:10.8290034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:10.8290735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:10.9571575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4cl7on9 2022-08-17T13:07:10.9572404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4cl7on9/_remote_module_non_scriptable.py 2022-08-17T13:07:10.9602586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1w056ot 2022-08-17T13:07:10.9603616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1w056ot/_remote_module_non_scriptable.py 2022-08-17T13:07:10.9759806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub_o7lux 2022-08-17T13:07:10.9760936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub_o7lux/_remote_module_non_scriptable.py 2022-08-17T13:07:10.9856468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenampcnj 2022-08-17T13:07:10.9857638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenampcnj/_remote_module_non_scriptable.py 2022-08-17T13:07:11.2187041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:11.2316894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:11.2407316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:11.2502081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:11.9459313Z ok (2.477s) 2022-08-17T13:07:11.9459573Z 2022-08-17T13:07:11.9460101Z ---------------------------------------------------------------------- 2022-08-17T13:07:11.9460390Z Ran 1 test in 2.477s 2022-08-17T13:07:11.9460510Z 2022-08-17T13:07:11.9460574Z OK 2022-08-17T13:07:11.9460669Z 2022-08-17T13:07:11.9460765Z Generating XML reports... 2022-08-17T13:07:11.9497191Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130709.xml 2022-08-17T13:07:13.2644191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:13.2644593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:13.2647104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:13.2647728Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:13.4212666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs0htjbx 2022-08-17T13:07:13.4215446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs0htjbx/_remote_module_non_scriptable.py 2022-08-17T13:07:13.6840092Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:13.6850486Z 2022-08-17T13:07:13.6850635Z Running tests... 2022-08-17T13:07:13.6851148Z ---------------------------------------------------------------------- 2022-08-17T13:07:13.9459554Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27284 2022-08-17T13:07:13.9489383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27285 2022-08-17T13:07:13.9519684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27286 2022-08-17T13:07:13.9550976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27287 2022-08-17T13:07:14.9347898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:14.9348554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:14.9349491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:14.9350120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:14.9745709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:14.9746414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:14.9749650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:14.9750399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:14.9966021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:14.9966709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:14.9970250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:14.9970938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:15.0284607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:15.0285137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:15.0288010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:15.0288603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:15.0917968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmbq8z5t 2022-08-17T13:07:15.0919055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmbq8z5t/_remote_module_non_scriptable.py 2022-08-17T13:07:15.1333936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnejedphz 2022-08-17T13:07:15.1334767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnejedphz/_remote_module_non_scriptable.py 2022-08-17T13:07:15.1566762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpzqj8q7 2022-08-17T13:07:15.1567545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpzqj8q7/_remote_module_non_scriptable.py 2022-08-17T13:07:15.1862643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wy5wf6s 2022-08-17T13:07:15.1863471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wy5wf6s/_remote_module_non_scriptable.py 2022-08-17T13:07:15.3527983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:15.3949579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:15.4193788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:15.4558026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:16.1611456Z ok (2.476s) 2022-08-17T13:07:16.1611722Z 2022-08-17T13:07:16.1612190Z ---------------------------------------------------------------------- 2022-08-17T13:07:16.1612473Z Ran 1 test in 2.476s 2022-08-17T13:07:16.1612597Z 2022-08-17T13:07:16.1612648Z OK 2022-08-17T13:07:16.1612743Z 2022-08-17T13:07:16.1612838Z Generating XML reports... 2022-08-17T13:07:16.1646022Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130713.xml 2022-08-17T13:07:17.4768513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:17.4769225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:17.4772638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:17.4773278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:17.6292372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lvkioqi 2022-08-17T13:07:17.6295349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lvkioqi/_remote_module_non_scriptable.py 2022-08-17T13:07:17.8978764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:17.8990485Z 2022-08-17T13:07:17.8990961Z Running tests... 2022-08-17T13:07:17.8991398Z ---------------------------------------------------------------------- 2022-08-17T13:07:18.1643366Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27519 2022-08-17T13:07:18.1672497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27520 2022-08-17T13:07:18.1700778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27521 2022-08-17T13:07:18.1731947Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27522 2022-08-17T13:07:19.1927269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:19.1927923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:19.1931156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:19.1931881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:19.1949989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:19.1950612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:19.1954118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:19.1954917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:19.2142196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:19.2142818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:19.2145489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:19.2146104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:19.2371300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:19.2371935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:19.2374856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:19.2375617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:19.3515311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wr9fuj4 2022-08-17T13:07:19.3516444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wr9fuj4/_remote_module_non_scriptable.py 2022-08-17T13:07:19.3544408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ygqrrnk 2022-08-17T13:07:19.3546479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ygqrrnk/_remote_module_non_scriptable.py 2022-08-17T13:07:19.3742357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel8hjq7t 2022-08-17T13:07:19.3743655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel8hjq7t/_remote_module_non_scriptable.py 2022-08-17T13:07:19.3950188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1_p3_nh 2022-08-17T13:07:19.3951040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1_p3_nh/_remote_module_non_scriptable.py 2022-08-17T13:07:19.6150257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:19.6378498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:19.6405247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:19.6594047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:22.3824475Z ok (4.483s) 2022-08-17T13:07:22.3824700Z 2022-08-17T13:07:22.3825155Z ---------------------------------------------------------------------- 2022-08-17T13:07:22.3825409Z Ran 1 test in 4.483s 2022-08-17T13:07:22.3825531Z 2022-08-17T13:07:22.3825600Z OK 2022-08-17T13:07:22.3825697Z 2022-08-17T13:07:22.3826046Z Generating XML reports... 2022-08-17T13:07:22.3859563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130717.xml 2022-08-17T13:07:23.6936862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:23.6937261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:23.6939597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:23.6939957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:23.8439648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49ukg30v 2022-08-17T13:07:23.8442763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49ukg30v/_remote_module_non_scriptable.py 2022-08-17T13:07:24.1065920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:24.1076791Z 2022-08-17T13:07:24.1076941Z Running tests... 2022-08-17T13:07:24.1077299Z ---------------------------------------------------------------------- 2022-08-17T13:07:24.3687161Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27742 2022-08-17T13:07:24.3716073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27743 2022-08-17T13:07:24.3744370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27744 2022-08-17T13:07:24.3775992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27745 2022-08-17T13:07:25.4010396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:25.4011016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:25.4013707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:25.4014320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:25.4159226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:25.4159894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:25.4162276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:25.4162931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:25.4259709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:25.4260441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:25.4264258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:25.4264967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:25.4301200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:25.4301859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:25.4305050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:25.4305696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:25.5609494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptimy70t_ 2022-08-17T13:07:25.5610399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptimy70t_/_remote_module_non_scriptable.py 2022-08-17T13:07:25.5752494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd754im5g 2022-08-17T13:07:25.5753558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd754im5g/_remote_module_non_scriptable.py 2022-08-17T13:07:25.5848677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0gy9ld0 2022-08-17T13:07:25.5849952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0gy9ld0/_remote_module_non_scriptable.py 2022-08-17T13:07:25.5871049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa1rietj 2022-08-17T13:07:25.5872410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa1rietj/_remote_module_non_scriptable.py 2022-08-17T13:07:25.8195857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:25.8352422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:25.8436289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:25.8438333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:28.4865268Z ok (4.378s) 2022-08-17T13:07:28.4865522Z 2022-08-17T13:07:28.4866003Z ---------------------------------------------------------------------- 2022-08-17T13:07:28.4866390Z Ran 1 test in 4.379s 2022-08-17T13:07:28.4866585Z 2022-08-17T13:07:28.4866696Z OK 2022-08-17T13:07:28.4866851Z 2022-08-17T13:07:28.4867001Z Generating XML reports... 2022-08-17T13:07:28.4900083Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130724.xml 2022-08-17T13:07:29.7689096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:29.7689496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:29.7691877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:29.7692241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:29.9198807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3n6t34h 2022-08-17T13:07:29.9202259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3n6t34h/_remote_module_non_scriptable.py 2022-08-17T13:07:30.1886850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:30.1898402Z 2022-08-17T13:07:30.1898526Z Running tests... 2022-08-17T13:07:30.1898956Z ---------------------------------------------------------------------- 2022-08-17T13:07:30.4526163Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27965 2022-08-17T13:07:30.4553314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27966 2022-08-17T13:07:30.4581445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27967 2022-08-17T13:07:30.4613672Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27968 2022-08-17T13:07:31.4666983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:31.4667678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:31.4670768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:31.4671482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:31.4740779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:31.4741425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:31.4744901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:31.4745584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:31.5053743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:31.5054395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:31.5057723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:31.5058390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:31.5099810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:31.5100532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:31.5103164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:31.5103960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:31.6261438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_nxuuuh 2022-08-17T13:07:31.6262202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_nxuuuh/_remote_module_non_scriptable.py 2022-08-17T13:07:31.6327655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm5gpsyo 2022-08-17T13:07:31.6330193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm5gpsyo/_remote_module_non_scriptable.py 2022-08-17T13:07:31.6649210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi1clhr9 2022-08-17T13:07:31.6650341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi1clhr9/_remote_module_non_scriptable.py 2022-08-17T13:07:31.6671988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dw2o8cd 2022-08-17T13:07:31.6673733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dw2o8cd/_remote_module_non_scriptable.py 2022-08-17T13:07:31.8859453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:31.8911610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:31.9266013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:31.9266728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:44.2057148Z [W tensorpipe_agent.cpp:940] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:07:44.2150195Z [W tensorpipe_agent.cpp:940] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:07:45.1483079Z [W tensorpipe_agent.cpp:625] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down 2022-08-17T13:07:45.1730620Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down 2022-08-17T13:07:45.1810574Z [W tensorpipe_agent.cpp:625] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down 2022-08-17T13:07:45.1811270Z [W tensorpipe_agent.cpp:625] RPC agent for worker3 won't send response to request #28 to worker2, as the agent is shutting down 2022-08-17T13:07:45.5876595Z ok (15.397s) 2022-08-17T13:07:45.5876826Z 2022-08-17T13:07:45.5877190Z ---------------------------------------------------------------------- 2022-08-17T13:07:45.5877521Z Ran 1 test in 15.398s 2022-08-17T13:07:45.5877747Z 2022-08-17T13:07:45.5877885Z OK 2022-08-17T13:07:45.5878055Z 2022-08-17T13:07:45.5878205Z Generating XML reports... 2022-08-17T13:07:45.5912418Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130730.xml 2022-08-17T13:07:46.9109012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:46.9109404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:46.9112002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:46.9112497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:47.0625141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgukanhk 2022-08-17T13:07:47.0628504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgukanhk/_remote_module_non_scriptable.py 2022-08-17T13:07:47.3266815Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:47.3277594Z 2022-08-17T13:07:47.3277715Z Running tests... 2022-08-17T13:07:47.3278353Z ---------------------------------------------------------------------- 2022-08-17T13:07:47.5910601Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28188 2022-08-17T13:07:47.5938740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28189 2022-08-17T13:07:47.5966640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28190 2022-08-17T13:07:47.5997283Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28191 2022-08-17T13:07:48.6800909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:48.6801562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:48.6804735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:48.6805449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:48.6828376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:48.6829008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:48.6832947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:48.6833606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:48.7068600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:48.7069602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:48.7071357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:48.7072027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:48.7100728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:48.7101383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:48.7106097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:48.7106763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:48.8410865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbu02t86d 2022-08-17T13:07:48.8411636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbu02t86d/_remote_module_non_scriptable.py 2022-08-17T13:07:48.8432181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kexpknj 2022-08-17T13:07:48.8433478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kexpknj/_remote_module_non_scriptable.py 2022-08-17T13:07:48.8679468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8jzi975 2022-08-17T13:07:48.8680207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8jzi975/_remote_module_non_scriptable.py 2022-08-17T13:07:48.8700871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vtuh1n9 2022-08-17T13:07:48.8702299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vtuh1n9/_remote_module_non_scriptable.py 2022-08-17T13:07:49.1028788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:49.1029939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:49.1315127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:49.1328117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:49.8058974Z ok (2.478s) 2022-08-17T13:07:49.8059236Z 2022-08-17T13:07:49.8059599Z ---------------------------------------------------------------------- 2022-08-17T13:07:49.8059855Z Ran 1 test in 2.478s 2022-08-17T13:07:49.8059977Z 2022-08-17T13:07:49.8060041Z OK 2022-08-17T13:07:49.8060139Z 2022-08-17T13:07:49.8060236Z Generating XML reports... 2022-08-17T13:07:49.8093852Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130747.xml 2022-08-17T13:07:51.1074908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:51.1075321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:51.1077636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:51.1078004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:51.2603780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqjf8wmq 2022-08-17T13:07:51.2607776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqjf8wmq/_remote_module_non_scriptable.py 2022-08-17T13:07:51.5282746Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:51.5293541Z 2022-08-17T13:07:51.5293659Z Running tests... 2022-08-17T13:07:51.5294124Z ---------------------------------------------------------------------- 2022-08-17T13:07:51.7987308Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28423 2022-08-17T13:07:51.8015528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28424 2022-08-17T13:07:51.8044681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28425 2022-08-17T13:07:51.8077657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28426 2022-08-17T13:07:52.8658567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:52.8659213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:52.8662037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:52.8662708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:52.8738754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:52.8739410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:52.8742832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:52.8743529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:52.8971996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:52.8972670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:52.8974979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:52.8975633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:52.8976445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:52.8977104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:52.8977941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:52.8978584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:53.0277219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifkgwaj2 2022-08-17T13:07:53.0277959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifkgwaj2/_remote_module_non_scriptable.py 2022-08-17T13:07:53.0348375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxjsxgeb 2022-08-17T13:07:53.0349543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxjsxgeb/_remote_module_non_scriptable.py 2022-08-17T13:07:53.0569491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlqqy53f 2022-08-17T13:07:53.0570227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8c92h53 2022-08-17T13:07:53.0570955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlqqy53f/_remote_module_non_scriptable.py 2022-08-17T13:07:53.0571668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8c92h53/_remote_module_non_scriptable.py 2022-08-17T13:07:53.2909917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:53.2950933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:53.3183409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:53.3184297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:54.1138885Z ok (2.584s) 2022-08-17T13:07:54.1139149Z 2022-08-17T13:07:54.1139671Z ---------------------------------------------------------------------- 2022-08-17T13:07:54.1140040Z Ran 1 test in 2.584s 2022-08-17T13:07:54.1140165Z 2022-08-17T13:07:54.1140498Z OK 2022-08-17T13:07:54.1140596Z 2022-08-17T13:07:54.1140699Z Generating XML reports... 2022-08-17T13:07:54.1175003Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130751.xml 2022-08-17T13:07:55.4294658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:55.4295089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:55.4297317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:55.4297690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:55.5795225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3o8yt6c 2022-08-17T13:07:55.5798126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3o8yt6c/_remote_module_non_scriptable.py 2022-08-17T13:07:55.8419904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:07:55.8430621Z 2022-08-17T13:07:55.8430749Z Running tests... 2022-08-17T13:07:55.8431336Z ---------------------------------------------------------------------- 2022-08-17T13:07:56.1027176Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28658 2022-08-17T13:07:56.1055374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28659 2022-08-17T13:07:56.1083833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28660 2022-08-17T13:07:56.1115059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28661 2022-08-17T13:07:57.2289656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:57.2290295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:57.2293414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:57.2294051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:57.2327182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:57.2327862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:57.2331594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:57.2332202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:57.2561587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:57.2562267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:57.2565729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:57.2566586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:57.2598027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:57.2598671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:57.2602538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:57.2603191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:57.3904311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqy99xas 2022-08-17T13:07:57.3905325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqy99xas/_remote_module_non_scriptable.py 2022-08-17T13:07:57.3923207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjfi76ew 2022-08-17T13:07:57.3924661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjfi76ew/_remote_module_non_scriptable.py 2022-08-17T13:07:57.4149258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2j0f463 2022-08-17T13:07:57.4150655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2j0f463/_remote_module_non_scriptable.py 2022-08-17T13:07:57.4175367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliorvgkh 2022-08-17T13:07:57.4176985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliorvgkh/_remote_module_non_scriptable.py 2022-08-17T13:07:57.6516706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:07:57.6520662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:07:57.6760965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:07:57.6782740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:07:58.5179408Z ok (2.675s) 2022-08-17T13:07:58.5179666Z 2022-08-17T13:07:58.5180093Z ---------------------------------------------------------------------- 2022-08-17T13:07:58.5180641Z Ran 1 test in 2.675s 2022-08-17T13:07:58.5180756Z 2022-08-17T13:07:58.5180823Z OK 2022-08-17T13:07:58.5180919Z 2022-08-17T13:07:58.5181014Z Generating XML reports... 2022-08-17T13:07:58.5214288Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130755.xml 2022-08-17T13:07:59.8379733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:07:59.8380134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:07:59.8382268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:07:59.8382677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:07:59.9899276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzw6w07t 2022-08-17T13:07:59.9901939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzw6w07t/_remote_module_non_scriptable.py 2022-08-17T13:08:00.2545476Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:00.2556496Z 2022-08-17T13:08:00.2556713Z Running tests... 2022-08-17T13:08:00.2557217Z ---------------------------------------------------------------------- 2022-08-17T13:08:00.5165697Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28893 2022-08-17T13:08:00.5193764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28894 2022-08-17T13:08:00.5223340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28895 2022-08-17T13:08:00.5255754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28896 2022-08-17T13:08:01.5733079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:01.5733730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:01.5735878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:01.5736481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:01.5747695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:01.5748300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:01.5752919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:01.5753895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:01.6139610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:01.6140254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:01.6143000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:01.6143708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:01.6232488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:01.6233033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:01.6235792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:01.6236347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:01.7373582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ge927j7 2022-08-17T13:08:01.7374756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ge927j7/_remote_module_non_scriptable.py 2022-08-17T13:08:01.7375469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndrb5d8q 2022-08-17T13:08:01.7376161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndrb5d8q/_remote_module_non_scriptable.py 2022-08-17T13:08:01.7758609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa14_dnc4 2022-08-17T13:08:01.7759393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa14_dnc4/_remote_module_non_scriptable.py 2022-08-17T13:08:01.7857909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yjohlbm 2022-08-17T13:08:01.7859181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yjohlbm/_remote_module_non_scriptable.py 2022-08-17T13:08:02.0004736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:02.0014103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:02.0410032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:02.0552947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:02.7314985Z ok (2.476s) 2022-08-17T13:08:02.7315205Z 2022-08-17T13:08:02.7315749Z ---------------------------------------------------------------------- 2022-08-17T13:08:02.7316178Z Ran 1 test in 2.476s 2022-08-17T13:08:02.7316299Z 2022-08-17T13:08:02.7316364Z OK 2022-08-17T13:08:02.7316458Z 2022-08-17T13:08:02.7316541Z Generating XML reports... 2022-08-17T13:08:02.7350219Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130800.xml 2022-08-17T13:08:04.0395528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:04.0396155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:04.0398407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:04.0399036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:04.1984592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpof4jrx6_ 2022-08-17T13:08:04.1988278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpof4jrx6_/_remote_module_non_scriptable.py 2022-08-17T13:08:04.4638604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:04.4649873Z 2022-08-17T13:08:04.4650478Z Running tests... 2022-08-17T13:08:04.4651159Z ---------------------------------------------------------------------- 2022-08-17T13:08:04.7260813Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29128 2022-08-17T13:08:04.7289469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29129 2022-08-17T13:08:04.7317280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29130 2022-08-17T13:08:04.7349954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29131 2022-08-17T13:08:05.7579703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:05.7580409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:05.7583205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:05.7583870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:05.7795535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:05.7796187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:05.7797864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:05.7798474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:05.7799186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:05.7799824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:05.7802617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:05.7803314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:05.7809479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:05.7810144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:05.7815208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:05.7815836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:05.9162603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkpbwz4f 2022-08-17T13:08:05.9163478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkpbwz4f/_remote_module_non_scriptable.py 2022-08-17T13:08:05.9391503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvduq8yr 2022-08-17T13:08:05.9392285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvduq8yr/_remote_module_non_scriptable.py 2022-08-17T13:08:05.9399394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_9lo5ur 2022-08-17T13:08:05.9401457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_9lo5ur/_remote_module_non_scriptable.py 2022-08-17T13:08:05.9418980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiv1ng88 2022-08-17T13:08:05.9421438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiv1ng88/_remote_module_non_scriptable.py 2022-08-17T13:08:06.1767711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:06.2005575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:06.2025435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:06.2049500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:06.9411431Z ok (2.476s) 2022-08-17T13:08:06.9411700Z 2022-08-17T13:08:06.9412207Z ---------------------------------------------------------------------- 2022-08-17T13:08:06.9412495Z Ran 1 test in 2.476s 2022-08-17T13:08:06.9412620Z 2022-08-17T13:08:06.9412686Z OK 2022-08-17T13:08:06.9412783Z 2022-08-17T13:08:06.9412879Z Generating XML reports... 2022-08-17T13:08:06.9445323Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130804.xml 2022-08-17T13:08:08.2589564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:08.2589957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:08.2592234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:08.2592664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:08.4143856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprn68iw4v 2022-08-17T13:08:08.4147205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprn68iw4v/_remote_module_non_scriptable.py 2022-08-17T13:08:08.6826619Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:08.6838708Z 2022-08-17T13:08:08.6838843Z Running tests... 2022-08-17T13:08:08.6839320Z ---------------------------------------------------------------------- 2022-08-17T13:08:08.9540521Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29363 2022-08-17T13:08:08.9569414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29364 2022-08-17T13:08:08.9599361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29365 2022-08-17T13:08:08.9632823Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29366 2022-08-17T13:08:09.9826593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:09.9827261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:09.9864726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:09.9865393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:09.9948808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:09.9949427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:09.9953225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:09.9953835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:10.0027655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:10.0028300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:10.0031938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:10.0032565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:10.0245470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:10.0246346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:10.0248960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:10.0249666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:10.1438557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugtj71lk 2022-08-17T13:08:10.1439823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugtj71lk/_remote_module_non_scriptable.py 2022-08-17T13:08:10.1534641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wu_fnet 2022-08-17T13:08:10.1536118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wu_fnet/_remote_module_non_scriptable.py 2022-08-17T13:08:10.1628255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19998_po 2022-08-17T13:08:10.1629791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19998_po/_remote_module_non_scriptable.py 2022-08-17T13:08:10.1819270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77zqosmk 2022-08-17T13:08:10.1820192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77zqosmk/_remote_module_non_scriptable.py 2022-08-17T13:08:10.4068436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:10.4257026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:10.4301290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:10.4425478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:11.1693235Z ok (2.485s) 2022-08-17T13:08:11.1693388Z 2022-08-17T13:08:11.1693847Z ---------------------------------------------------------------------- 2022-08-17T13:08:11.1694133Z Ran 1 test in 2.485s 2022-08-17T13:08:11.1694254Z 2022-08-17T13:08:11.1694318Z OK 2022-08-17T13:08:11.1694416Z 2022-08-17T13:08:11.1694570Z Generating XML reports... 2022-08-17T13:08:11.1727353Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130808.xml 2022-08-17T13:08:12.5029840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:12.5030318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:12.5032755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:12.5033140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:12.6546966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3whmoc5 2022-08-17T13:08:12.6549557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3whmoc5/_remote_module_non_scriptable.py 2022-08-17T13:08:12.9186983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:12.9197764Z 2022-08-17T13:08:12.9197870Z Running tests... 2022-08-17T13:08:12.9198450Z ---------------------------------------------------------------------- 2022-08-17T13:08:13.1823701Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29598 2022-08-17T13:08:13.1851171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29599 2022-08-17T13:08:13.1879250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29600 2022-08-17T13:08:13.1912670Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29601 2022-08-17T13:08:14.2788629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:14.2789259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:14.2791483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:14.2792136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:14.2832750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:14.2833716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:14.2836399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:14.2837034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:14.3131758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:14.3132388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:14.3135461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:14.3136081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:14.3721646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:14.3722312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:14.3723726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:14.3724686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:14.4390799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoew6ratu 2022-08-17T13:08:14.4391628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoew6ratu/_remote_module_non_scriptable.py 2022-08-17T13:08:14.4455331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe89xfrfq 2022-08-17T13:08:14.4456630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe89xfrfq/_remote_module_non_scriptable.py 2022-08-17T13:08:14.4760243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn89p0q4c 2022-08-17T13:08:14.4761002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn89p0q4c/_remote_module_non_scriptable.py 2022-08-17T13:08:14.5320449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkmsy02b 2022-08-17T13:08:14.5321622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkmsy02b/_remote_module_non_scriptable.py 2022-08-17T13:08:14.7008924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:14.7328629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:14.7435503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:14.7957162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:15.3973072Z ok (2.477s) 2022-08-17T13:08:15.3973345Z 2022-08-17T13:08:15.3973804Z ---------------------------------------------------------------------- 2022-08-17T13:08:15.3974090Z Ran 1 test in 2.477s 2022-08-17T13:08:15.3974224Z 2022-08-17T13:08:15.3974290Z OK 2022-08-17T13:08:15.3974386Z 2022-08-17T13:08:15.3974480Z Generating XML reports... 2022-08-17T13:08:15.4008869Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130812.xml 2022-08-17T13:08:16.7073750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:16.7074142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:16.7076427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:16.7076794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:16.8636190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxv05faqw 2022-08-17T13:08:16.8639946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxv05faqw/_remote_module_non_scriptable.py 2022-08-17T13:08:17.1309972Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:17.1319830Z 2022-08-17T13:08:17.1319953Z Running tests... 2022-08-17T13:08:17.1320591Z ---------------------------------------------------------------------- 2022-08-17T13:08:17.3957057Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29865 2022-08-17T13:08:17.3985828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29866 2022-08-17T13:08:17.4014113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29867 2022-08-17T13:08:17.4045385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29868 2022-08-17T13:08:18.4587402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:18.4588077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:18.4591129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:18.4591769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:18.4655973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:18.4656629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:18.4657418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:18.4658031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:18.4659707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:18.4660367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:18.4661201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:18.4661846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:18.4809684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:18.4810319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:18.4812994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:18.4813647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:18.6178279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcox6ev5n 2022-08-17T13:08:18.6179514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcox6ev5n/_remote_module_non_scriptable.py 2022-08-17T13:08:18.6236292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg54rg5v8 2022-08-17T13:08:18.6238386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg54rg5v8/_remote_module_non_scriptable.py 2022-08-17T13:08:18.6258649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqstsey0 2022-08-17T13:08:18.6260242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqstsey0/_remote_module_non_scriptable.py 2022-08-17T13:08:18.6391584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmmcrnb3 2022-08-17T13:08:18.6392309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmmcrnb3/_remote_module_non_scriptable.py 2022-08-17T13:08:18.8803518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:18.8868037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:18.9083830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:18.9140901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:19.2769518Z Rank 0 finished testing 20 times in 0.13966655731201172 seconds. 2022-08-17T13:08:19.2934208Z Rank 3 finished testing 20 times in 0.07629847526550293 seconds. 2022-08-17T13:08:19.3278004Z Rank 1 finished testing 20 times in 0.09449410438537598 seconds. 2022-08-17T13:08:19.3390128Z Rank 2 finished testing 20 times in 0.129866361618042 seconds. 2022-08-17T13:08:19.7108090Z ok (2.578s) 2022-08-17T13:08:19.7108323Z 2022-08-17T13:08:19.7108796Z ---------------------------------------------------------------------- 2022-08-17T13:08:19.7109217Z Ran 1 test in 2.579s 2022-08-17T13:08:19.7109388Z 2022-08-17T13:08:19.7109501Z OK 2022-08-17T13:08:19.7109658Z 2022-08-17T13:08:19.7109804Z Generating XML reports... 2022-08-17T13:08:19.7144091Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130817.xml 2022-08-17T13:08:21.0160674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:21.0161086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:21.0163133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:21.0163525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:21.1670680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkl3td25 2022-08-17T13:08:21.1674036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkl3td25/_remote_module_non_scriptable.py 2022-08-17T13:08:21.4308553Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:21.4319496Z 2022-08-17T13:08:21.4319659Z Running tests... 2022-08-17T13:08:21.4320269Z ---------------------------------------------------------------------- 2022-08-17T13:08:21.6992435Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30100 2022-08-17T13:08:21.7021132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30101 2022-08-17T13:08:21.7050292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30102 2022-08-17T13:08:21.7082473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30103 2022-08-17T13:08:22.7844995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:22.7845607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:22.7848619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:22.7849322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:22.8140227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:22.8140932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:22.8144011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:22.8144678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:22.9097050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:22.9097737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:22.9100701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:22.9101650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:22.9427890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfg47n4o3 2022-08-17T13:08:22.9429162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfg47n4o3/_remote_module_non_scriptable.py 2022-08-17T13:08:22.9463293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:22.9464072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:22.9468534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:22.9469421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:22.9760576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bv_oong 2022-08-17T13:08:22.9761228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bv_oong/_remote_module_non_scriptable.py 2022-08-17T13:08:23.0710227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb6vyckl 2022-08-17T13:08:23.0710983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb6vyckl/_remote_module_non_scriptable.py 2022-08-17T13:08:23.1038358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sl5k68p 2022-08-17T13:08:23.1039175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sl5k68p/_remote_module_non_scriptable.py 2022-08-17T13:08:23.2025214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:23.2368723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:23.3352355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:23.3688096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:23.7136756Z ok (2.281s) 2022-08-17T13:08:23.7137025Z 2022-08-17T13:08:23.7137357Z ---------------------------------------------------------------------- 2022-08-17T13:08:23.7137619Z Ran 1 test in 2.282s 2022-08-17T13:08:23.7137726Z 2022-08-17T13:08:23.7137790Z OK 2022-08-17T13:08:23.7137886Z 2022-08-17T13:08:23.7137998Z Generating XML reports... 2022-08-17T13:08:23.7172377Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130821.xml 2022-08-17T13:08:25.0308727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:25.0309112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:25.0311714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:25.0312096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:25.1823610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts56y883 2022-08-17T13:08:25.1827016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts56y883/_remote_module_non_scriptable.py 2022-08-17T13:08:25.4480066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:25.4491102Z 2022-08-17T13:08:25.4491205Z Running tests... 2022-08-17T13:08:25.4491631Z ---------------------------------------------------------------------- 2022-08-17T13:08:25.7114896Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30159 2022-08-17T13:08:25.7143211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30160 2022-08-17T13:08:25.7173259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30161 2022-08-17T13:08:25.7205346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30162 2022-08-17T13:08:26.7749194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:26.7749833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:26.7753228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:26.7753937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:26.7992876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:26.7993526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:26.7996345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:26.7997030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:26.8322387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:26.8322977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:26.8325722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:26.8326516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:26.8363853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:26.8364506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:26.8368590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:26.8369246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:26.9324130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5liukepg 2022-08-17T13:08:26.9324931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5liukepg/_remote_module_non_scriptable.py 2022-08-17T13:08:26.9583066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1wnugb0 2022-08-17T13:08:26.9584002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1wnugb0/_remote_module_non_scriptable.py 2022-08-17T13:08:26.9914989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqy6o9fe 2022-08-17T13:08:26.9915861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqy6o9fe/_remote_module_non_scriptable.py 2022-08-17T13:08:26.9935838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiwbvype 2022-08-17T13:08:26.9936963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiwbvype/_remote_module_non_scriptable.py 2022-08-17T13:08:27.1967649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:27.2411423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:27.2526854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:27.2543425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:27.8262504Z ok (2.377s) 2022-08-17T13:08:27.8262711Z 2022-08-17T13:08:27.8263109Z ---------------------------------------------------------------------- 2022-08-17T13:08:27.8263391Z Ran 1 test in 2.377s 2022-08-17T13:08:27.8263546Z 2022-08-17T13:08:27.8263644Z OK 2022-08-17T13:08:27.8263742Z 2022-08-17T13:08:27.8263838Z Generating XML reports... 2022-08-17T13:08:27.8296969Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130825.xml 2022-08-17T13:08:29.1452762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:29.1453406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:29.1455727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:29.1456129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:29.2988271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij1oze_h 2022-08-17T13:08:29.2991032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij1oze_h/_remote_module_non_scriptable.py 2022-08-17T13:08:29.5594386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:29.5604820Z 2022-08-17T13:08:29.5604968Z Running tests... 2022-08-17T13:08:29.5605433Z ---------------------------------------------------------------------- 2022-08-17T13:08:29.8217135Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30382 2022-08-17T13:08:29.8245263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30383 2022-08-17T13:08:29.8274943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30384 2022-08-17T13:08:29.8306521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30385 2022-08-17T13:08:30.8630339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:30.8630984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:30.8634110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:30.8634756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:30.8915774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:30.8916374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:30.8919338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:30.8920003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:30.9325633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:30.9326456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:30.9329388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:30.9329974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:30.9460901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:30.9461569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:30.9464214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:30.9464893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:31.0213279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ygjkhaw 2022-08-17T13:08:31.0214303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ygjkhaw/_remote_module_non_scriptable.py 2022-08-17T13:08:31.0525486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ip69drz 2022-08-17T13:08:31.0526537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ip69drz/_remote_module_non_scriptable.py 2022-08-17T13:08:31.0931537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4oaoruyk 2022-08-17T13:08:31.0932681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4oaoruyk/_remote_module_non_scriptable.py 2022-08-17T13:08:31.1080235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10j5j0eo 2022-08-17T13:08:31.1081458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10j5j0eo/_remote_module_non_scriptable.py 2022-08-17T13:08:31.2835634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:31.3358534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:31.3616689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:31.3798199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:31.4216912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:08:31.4217346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:08:31.4318090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:08:31.4319310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:08:31.4320591Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:31.4321431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:31.4321990Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:31.4322546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:31.8433602Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:31.8434565Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:31.8435469Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:31.8436084Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:31.8436631Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:31.8437212Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:08:32.2370589Z ok (2.676s) 2022-08-17T13:08:32.2370832Z 2022-08-17T13:08:32.2371311Z ---------------------------------------------------------------------- 2022-08-17T13:08:32.2371612Z Ran 1 test in 2.676s 2022-08-17T13:08:32.2371739Z 2022-08-17T13:08:32.2371829Z OK 2022-08-17T13:08:32.2371979Z 2022-08-17T13:08:32.2372514Z Generating XML reports... 2022-08-17T13:08:32.2406069Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130829.xml 2022-08-17T13:08:33.5320882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:33.5321539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:33.5326186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:33.5327219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:33.6832710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w0cel3q 2022-08-17T13:08:33.6835276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w0cel3q/_remote_module_non_scriptable.py 2022-08-17T13:08:33.9467639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:33.9480515Z 2022-08-17T13:08:33.9480885Z Running tests... 2022-08-17T13:08:33.9481317Z ---------------------------------------------------------------------- 2022-08-17T13:08:34.2124780Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30629 2022-08-17T13:08:34.2155034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30630 2022-08-17T13:08:34.2183162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30631 2022-08-17T13:08:34.2214427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30632 2022-08-17T13:08:35.2625661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:35.2626350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:35.2628879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:35.2629510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:35.2737267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:35.2737949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:35.2740942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:35.2741583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:35.2751498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:35.2752121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:35.2756276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:35.2756909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:35.2999697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:35.3000357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:35.3003390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:35.3004073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:35.4229680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5j05sntl 2022-08-17T13:08:35.4230539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5j05sntl/_remote_module_non_scriptable.py 2022-08-17T13:08:35.4338653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93knvtpp 2022-08-17T13:08:35.4340730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93knvtpp/_remote_module_non_scriptable.py 2022-08-17T13:08:35.4437740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7jgg4_t 2022-08-17T13:08:35.4439686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7jgg4_t/_remote_module_non_scriptable.py 2022-08-17T13:08:35.4580865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1dikq27 2022-08-17T13:08:35.4581797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1dikq27/_remote_module_non_scriptable.py 2022-08-17T13:08:35.6878875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:35.6985018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:35.7192115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:35.7222765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:35.7598051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:08:35.7699478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:08:35.7801430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:08:35.7802535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:35.7803202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:08:35.7804225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:35.7805049Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:35.7806153Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:08:36.1805386Z [W tensorpipe_agent.cpp:726] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:36.1813861Z [W tensorpipe_agent.cpp:726] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:36.1814913Z [W tensorpipe_agent.cpp:726] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259) 2022-08-17T13:08:36.1832286Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:36.1833229Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:36.1833934Z [W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:08:36.5276352Z ok (2.579s) 2022-08-17T13:08:36.5276853Z 2022-08-17T13:08:36.5277481Z ---------------------------------------------------------------------- 2022-08-17T13:08:36.5277922Z Ran 1 test in 2.580s 2022-08-17T13:08:36.5278118Z 2022-08-17T13:08:36.5278228Z OK 2022-08-17T13:08:36.5278394Z 2022-08-17T13:08:36.5278536Z Generating XML reports... 2022-08-17T13:08:36.5313155Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130833.xml 2022-08-17T13:08:37.8540614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:37.8541004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:37.8544142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:37.8544537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:38.0060915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7etncqww 2022-08-17T13:08:38.0064075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7etncqww/_remote_module_non_scriptable.py 2022-08-17T13:08:38.2692678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:38.2703660Z 2022-08-17T13:08:38.2704002Z Running tests... 2022-08-17T13:08:38.2704470Z ---------------------------------------------------------------------- 2022-08-17T13:08:38.5320195Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30876 2022-08-17T13:08:38.5347294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30877 2022-08-17T13:08:38.5375444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30878 2022-08-17T13:08:38.5405762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30879 2022-08-17T13:08:39.5508603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:39.5509262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:39.5512912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:39.5513606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:39.5954797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:39.5955471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:39.5958368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:39.5959166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:39.5965516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:39.5966338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:39.5971307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:39.5971927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:39.6198824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:39.6199462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:39.6203168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:39.6203845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:39.7276133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dc_nm33 2022-08-17T13:08:39.7276987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dc_nm33/_remote_module_non_scriptable.py 2022-08-17T13:08:39.7562859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdzg3pa2 2022-08-17T13:08:39.7564050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdzg3pa2/_remote_module_non_scriptable.py 2022-08-17T13:08:39.7604748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdi4w_uhz 2022-08-17T13:08:39.7606867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdi4w_uhz/_remote_module_non_scriptable.py 2022-08-17T13:08:39.7815085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto8nj0ib 2022-08-17T13:08:39.7816481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto8nj0ib/_remote_module_non_scriptable.py 2022-08-17T13:08:39.9893216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:40.0179029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:40.0237478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:08:40.0501852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:40.4461708Z ok (2.175s) 2022-08-17T13:08:40.4461955Z 2022-08-17T13:08:40.4462439Z ---------------------------------------------------------------------- 2022-08-17T13:08:40.4462861Z Ran 1 test in 2.176s 2022-08-17T13:08:40.4463034Z 2022-08-17T13:08:40.4463127Z OK 2022-08-17T13:08:40.4463274Z 2022-08-17T13:08:40.4463420Z Generating XML reports... 2022-08-17T13:08:40.4497317Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130838.xml 2022-08-17T13:08:41.7434329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:41.7434731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:41.7436797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:41.7437486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:41.8983100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl44eibh 2022-08-17T13:08:41.8985985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl44eibh/_remote_module_non_scriptable.py 2022-08-17T13:08:42.1626929Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:42.1637819Z 2022-08-17T13:08:42.1637990Z Running tests... 2022-08-17T13:08:42.1638437Z ---------------------------------------------------------------------- 2022-08-17T13:08:42.4293152Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30985 2022-08-17T13:08:42.4321263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30986 2022-08-17T13:08:42.4350198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30987 2022-08-17T13:08:43.4740596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:43.4741255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:43.4742946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:43.4743566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:43.4834156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:43.4834841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:43.4838831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:43.4839429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:43.4939248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:43.4939915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:43.4943643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:43.4944371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:43.6321052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgicmvihd 2022-08-17T13:08:43.6322198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgicmvihd/_remote_module_non_scriptable.py 2022-08-17T13:08:43.6435317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph273v_j0 2022-08-17T13:08:43.6436864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph273v_j0/_remote_module_non_scriptable.py 2022-08-17T13:08:43.6529815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hc5io__ 2022-08-17T13:08:43.6531849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hc5io__/_remote_module_non_scriptable.py 2022-08-17T13:08:43.8941600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:43.9101578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:43.9171610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:44.5407266Z ok (2.377s) 2022-08-17T13:08:44.5407504Z 2022-08-17T13:08:44.5408061Z ---------------------------------------------------------------------- 2022-08-17T13:08:44.5408357Z Ran 1 test in 2.377s 2022-08-17T13:08:44.5408480Z 2022-08-17T13:08:44.5408532Z OK 2022-08-17T13:08:44.5408629Z 2022-08-17T13:08:44.5408723Z Generating XML reports... 2022-08-17T13:08:44.5444774Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130842.xml 2022-08-17T13:08:45.8434491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:45.8434918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:45.8436769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:45.8437148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:45.9936088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1mts2py 2022-08-17T13:08:45.9938930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1mts2py/_remote_module_non_scriptable.py 2022-08-17T13:08:46.2562799Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:46.2573905Z 2022-08-17T13:08:46.2574025Z Running tests... 2022-08-17T13:08:46.2574676Z ---------------------------------------------------------------------- 2022-08-17T13:08:46.5237260Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31155 2022-08-17T13:08:46.5265116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31156 2022-08-17T13:08:46.5294568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31157 2022-08-17T13:08:47.5280531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:47.5281171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:47.5282741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:47.5283297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:47.5417053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:47.5417688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:47.5420857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:47.5421251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:47.5596041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:47.5596437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:47.5599825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:47.5600189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:47.6842737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit_m5ug9 2022-08-17T13:08:47.6843302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit_m5ug9/_remote_module_non_scriptable.py 2022-08-17T13:08:47.6994609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr32ximk 2022-08-17T13:08:47.6996512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr32ximk/_remote_module_non_scriptable.py 2022-08-17T13:08:47.7151377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5stm9ykw 2022-08-17T13:08:47.7153164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5stm9ykw/_remote_module_non_scriptable.py 2022-08-17T13:08:47.9442839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:47.9612253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:47.9758975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:48.5347734Z ok (2.277s) 2022-08-17T13:08:48.5348042Z 2022-08-17T13:08:48.5348409Z ---------------------------------------------------------------------- 2022-08-17T13:08:48.5348673Z Ran 1 test in 2.277s 2022-08-17T13:08:48.5348797Z 2022-08-17T13:08:48.5348861Z OK 2022-08-17T13:08:48.5348957Z 2022-08-17T13:08:48.5349057Z Generating XML reports... 2022-08-17T13:08:48.5382290Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130846.xml 2022-08-17T13:08:49.8580842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:49.8581286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:49.8583713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:49.8584172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:50.0091597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy6_t0cc 2022-08-17T13:08:50.0095244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy6_t0cc/_remote_module_non_scriptable.py 2022-08-17T13:08:50.2738423Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent 2022-08-17T13:08:50.2748655Z 2022-08-17T13:08:50.2748758Z Running tests... 2022-08-17T13:08:50.2749284Z ---------------------------------------------------------------------- 2022-08-17T13:08:50.5387457Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31325 2022-08-17T13:08:50.5416472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31326 2022-08-17T13:08:50.5445568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31327 2022-08-17T13:08:51.5257717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:51.5258367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:51.5261243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:51.5261900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:51.5449522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:51.5450182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:51.5451258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:51.5451851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:51.5452978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:51.5453549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:51.5454306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:51.5454939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:51.6831342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukmbrz4g 2022-08-17T13:08:51.6832370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukmbrz4g/_remote_module_non_scriptable.py 2022-08-17T13:08:51.7017013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffv32xw0 2022-08-17T13:08:51.7019401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffv32xw0/_remote_module_non_scriptable.py 2022-08-17T13:08:51.7026241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3z8h1zw1 2022-08-17T13:08:51.7028351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3z8h1zw1/_remote_module_non_scriptable.py 2022-08-17T13:08:51.9434150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:08:51.9611179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:51.9635119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:52.1708283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffv32xw0/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T13:08:52.1710148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3z8h1zw1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T13:08:52.1778597Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpffv32xw0/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-08-17T13:08:52.5496234Z ok (2.274s) 2022-08-17T13:08:52.5496460Z 2022-08-17T13:08:52.5496930Z ---------------------------------------------------------------------- 2022-08-17T13:08:52.5497361Z Ran 1 test in 2.275s 2022-08-17T13:08:52.5497537Z 2022-08-17T13:08:52.5497644Z OK 2022-08-17T13:08:52.5497808Z 2022-08-17T13:08:52.5497943Z Generating XML reports... 2022-08-17T13:08:52.5532160Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130850.xml 2022-08-17T13:08:53.1589758Z Running distributed/test_c10d_common ... [2022-08-17 13:08:53.158528] 2022-08-17T13:08:53.1590354Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:08:53.158616] 2022-08-17T13:08:54.2861486Z test_debug_level (__main__.CommTest) 2022-08-17T13:08:54.2862061Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-08-17T13:08:54.2862419Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-08-17T13:08:54.2862776Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-08-17T13:08:54.2863111Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-08-17T13:08:54.2863439Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-08-17T13:08:54.2863881Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-08-17T13:08:54.2864216Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-08-17T13:08:54.2864757Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-08-17T13:08:55.2201128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:55.2201526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:55.2203556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:55.2204072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:55.3710789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:08:55.3720420Z 2022-08-17T13:08:55.3720710Z Running tests... 2022-08-17T13:08:55.3721151Z ---------------------------------------------------------------------- 2022-08-17T13:08:55.6339256Z test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31505 2022-08-17T13:08:55.6367068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31506 2022-08-17T13:08:56.5934372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:56.5935032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:56.5938334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:56.5939021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:56.5969047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:56.5969703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:56.5973136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:56.5973797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:56.7456370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:08:56.7492467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:08:57.0405381Z ok (1.668s) 2022-08-17T13:08:57.0405616Z 2022-08-17T13:08:57.0406236Z ---------------------------------------------------------------------- 2022-08-17T13:08:57.0406709Z Ran 1 test in 1.668s 2022-08-17T13:08:57.0406906Z 2022-08-17T13:08:57.0407018Z OK 2022-08-17T13:08:57.0407188Z 2022-08-17T13:08:57.0407355Z Generating XML reports... 2022-08-17T13:08:57.0455016Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220817130855.xml 2022-08-17T13:08:58.3321884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:58.3322281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:58.3324457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:58.3324973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:08:58.4817000Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:08:58.4826450Z 2022-08-17T13:08:58.4826580Z Running tests... 2022-08-17T13:08:58.4827191Z ---------------------------------------------------------------------- 2022-08-17T13:08:58.7379768Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.255s) 2022-08-17T13:08:58.7380145Z 2022-08-17T13:08:58.7380641Z ---------------------------------------------------------------------- 2022-08-17T13:08:58.7380923Z Ran 1 test in 0.255s 2022-08-17T13:08:58.7381044Z 2022-08-17T13:08:58.7381109Z OK 2022-08-17T13:08:58.7381193Z 2022-08-17T13:08:58.7381282Z Generating XML reports... 2022-08-17T13:08:58.7415080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130858.xml 2022-08-17T13:08:59.9704226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:08:59.9704984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:08:59.9707153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:08:59.9707772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:00.1207156Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:00.1217047Z 2022-08-17T13:09:00.1217507Z Running tests... 2022-08-17T13:09:00.1217949Z ---------------------------------------------------------------------- 2022-08-17T13:09:00.3797842Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.258s) 2022-08-17T13:09:00.3798261Z 2022-08-17T13:09:00.3798775Z ---------------------------------------------------------------------- 2022-08-17T13:09:00.3799240Z Ran 1 test in 0.258s 2022-08-17T13:09:00.3799442Z 2022-08-17T13:09:00.3799561Z OK 2022-08-17T13:09:00.3799732Z 2022-08-17T13:09:00.3799895Z Generating XML reports... 2022-08-17T13:09:00.3831428Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130900.xml 2022-08-17T13:09:01.6272596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:01.6272993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:01.6275398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:01.6275777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:01.7807925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:01.7817941Z 2022-08-17T13:09:01.7818080Z Running tests... 2022-08-17T13:09:01.7818668Z ---------------------------------------------------------------------- 2022-08-17T13:09:02.0429479Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.261s) 2022-08-17T13:09:02.0429883Z 2022-08-17T13:09:02.0430395Z ---------------------------------------------------------------------- 2022-08-17T13:09:02.0430680Z Ran 1 test in 0.261s 2022-08-17T13:09:02.0430801Z 2022-08-17T13:09:02.0430867Z OK 2022-08-17T13:09:02.0430949Z 2022-08-17T13:09:02.0431038Z Generating XML reports... 2022-08-17T13:09:02.0463877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130901.xml 2022-08-17T13:09:03.2963887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:03.2964458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:03.2966772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:03.2967406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:03.4480867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:03.4491124Z 2022-08-17T13:09:03.4491533Z Running tests... 2022-08-17T13:09:03.4491930Z ---------------------------------------------------------------------- 2022-08-17T13:09:03.7047926Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.255s) 2022-08-17T13:09:03.7048332Z 2022-08-17T13:09:03.7048793Z ---------------------------------------------------------------------- 2022-08-17T13:09:03.7049068Z Ran 1 test in 0.256s 2022-08-17T13:09:03.7049175Z 2022-08-17T13:09:03.7049238Z OK 2022-08-17T13:09:03.7049335Z 2022-08-17T13:09:03.7049629Z Generating XML reports... 2022-08-17T13:09:03.7079815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130903.xml 2022-08-17T13:09:04.9341772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:04.9342184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:04.9344604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:04.9344980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:05.0855601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:05.0865374Z 2022-08-17T13:09:05.0865607Z Running tests... 2022-08-17T13:09:05.0866021Z ---------------------------------------------------------------------- 2022-08-17T13:09:05.3472037Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31578 2022-08-17T13:09:05.3502653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31579 2022-08-17T13:09:05.3545740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31580 2022-08-17T13:09:05.3582378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31581 2022-08-17T13:09:06.3561925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:06.3562570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:06.3565428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:06.3566362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:06.3790591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:06.3791284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:06.3793465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:06.3794493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:06.4141958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:06.4142604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:06.4144347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:06.4144964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:06.4164101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:06.4164727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:06.4168735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:06.4169405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:06.5321579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:09:06.5373039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:09:06.5728312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:06.5759612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:06.8623962Z ok (1.776s) 2022-08-17T13:09:06.8624223Z 2022-08-17T13:09:06.8624681Z ---------------------------------------------------------------------- 2022-08-17T13:09:06.8625355Z Ran 1 test in 1.776s 2022-08-17T13:09:06.8625547Z 2022-08-17T13:09:06.8625644Z OK 2022-08-17T13:09:06.8625791Z 2022-08-17T13:09:06.8625936Z Generating XML reports... 2022-08-17T13:09:06.8673621Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130905.xml 2022-08-17T13:09:08.1690208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:08.1690594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:08.1693080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:08.1693457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:08.3209013Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:08.3218916Z 2022-08-17T13:09:08.3219027Z Running tests... 2022-08-17T13:09:08.3219532Z ---------------------------------------------------------------------- 2022-08-17T13:09:08.5866727Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31633 2022-08-17T13:09:08.5894825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31634 2022-08-17T13:09:08.5929479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31635 2022-08-17T13:09:08.5967839Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31636 2022-08-17T13:09:09.6409653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:09.6410335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:09.6412495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:09.6413139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:09.6538715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:09.6539357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:09.6542351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:09.6543033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:09.6683777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:09.6684453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:09.6687561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:09.6688241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:09.7055470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:09.7056057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:09.7069926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:09.7070588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:09.8005029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:09:09.8010432Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-08-17T13:09:09.8133984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:09.8250410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:09.8258093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:09:09.8602740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:09:09.8611665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:09:10.8019042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:09:10.8070673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:09:10.8071807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:10.8119266Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:10.8120527Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:10.8165821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:11.1026039Z ok (2.780s) 2022-08-17T13:09:11.1026302Z 2022-08-17T13:09:11.1027095Z ---------------------------------------------------------------------- 2022-08-17T13:09:11.1027367Z Ran 1 test in 2.781s 2022-08-17T13:09:11.1027475Z 2022-08-17T13:09:11.1027541Z OK 2022-08-17T13:09:11.1027639Z 2022-08-17T13:09:11.1027736Z Generating XML reports... 2022-08-17T13:09:11.1076818Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130908.xml 2022-08-17T13:09:12.3747824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:12.3748209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:12.3750736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:12.3751120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:12.5256092Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:12.5265781Z 2022-08-17T13:09:12.5266182Z Running tests... 2022-08-17T13:09:12.5266575Z ---------------------------------------------------------------------- 2022-08-17T13:09:12.7900296Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31693 2022-08-17T13:09:12.7928641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31694 2022-08-17T13:09:12.7963346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31695 2022-08-17T13:09:12.7999702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31696 2022-08-17T13:09:13.8662088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:13.8662698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:13.8665621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:13.8666272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:13.8748782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:13.8749435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:13.8752130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:13.8752818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:13.9220931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:13.9221792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:13.9223044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:13.9223596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:13.9605715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:13.9606554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:13.9608811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:13.9609405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:14.0224795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:09:14.0315154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:14.0797459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:14.1204236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:09:14.4045252Z ok (1.878s) 2022-08-17T13:09:14.4045419Z 2022-08-17T13:09:14.4046008Z ---------------------------------------------------------------------- 2022-08-17T13:09:14.4046280Z Ran 1 test in 1.878s 2022-08-17T13:09:14.4046402Z 2022-08-17T13:09:14.4048401Z OK 2022-08-17T13:09:14.4048639Z 2022-08-17T13:09:14.4048819Z Generating XML reports... 2022-08-17T13:09:14.4093423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130912.xml 2022-08-17T13:09:15.6887620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:15.6888017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:15.6890302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:15.6890730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:15.8427582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-08-17T13:09:15.8437201Z 2022-08-17T13:09:15.8437539Z Running tests... 2022-08-17T13:09:15.8437958Z ---------------------------------------------------------------------- 2022-08-17T13:09:16.1083501Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31748 2022-08-17T13:09:16.1111610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31749 2022-08-17T13:09:16.1143329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31750 2022-08-17T13:09:16.1170692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31751 2022-08-17T13:09:17.2083113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:17.2083754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:17.2086484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:17.2087098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:17.2714063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:17.2714725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:17.2716424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:17.2717391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:17.3126362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:17.3126995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:17.3127974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:17.3128739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:17.3129505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:17.3130132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:17.3130917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:17.3131468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:17.3653698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:17.4281601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:17.4289966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:09:17.4702903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:09:17.4705550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:09:17.4711933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:09:17.4713855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:09:17.4781749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:09:17.4783003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:17.4796434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:17.4814740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:17.4818178Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:09:17.8218483Z ok (1.978s) 2022-08-17T13:09:17.8218742Z 2022-08-17T13:09:17.8219252Z ---------------------------------------------------------------------- 2022-08-17T13:09:17.8219728Z Ran 1 test in 1.978s 2022-08-17T13:09:17.8219851Z 2022-08-17T13:09:17.8219902Z OK 2022-08-17T13:09:17.8220001Z 2022-08-17T13:09:17.8220097Z Generating XML reports... 2022-08-17T13:09:17.8266104Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130915.xml 2022-08-17T13:09:18.3797346Z Running distributed/test_c10d_gloo ... [2022-08-17 13:09:18.379340] 2022-08-17T13:09:18.3797977Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:09:18.379430] 2022-08-17T13:09:19.5183475Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>]> 2022-08-17T13:09:19.5185322Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-08-17T13:09:19.5185657Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-08-17T13:09:19.5186027Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-08-17T13:09:19.5186346Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-08-17T13:09:19.5186709Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-08-17T13:09:19.5187104Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-08-17T13:09:19.5187469Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-08-17T13:09:19.5187913Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-08-17T13:09:19.5192874Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-08-17T13:09:19.5197329Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5197714Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5198099Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5198460Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5198855Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5199264Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5199656Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5200018Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5200389Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5200772Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5201188Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5201585Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5201977Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5202353Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5202694Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5203042Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5203386Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5203711Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5204050Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5204425Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5204792Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5205140Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5205521Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5206182Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5206742Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5207249Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5207693Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5208131Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5208573Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5209012Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5209502Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5209941Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5210372Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5210798Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5211227Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5211661Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5212244Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-08-17T13:09:19.5212637Z 2022-08-17T13:09:19.5218413Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-08-17T13:09:19.5224123Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5224527Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5224902Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5225308Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5225801Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5226216Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5226630Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5227027Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5227417Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5227791Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5228223Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5228670Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5229066Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5229471Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5229885Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5230286Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5230709Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5231136Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5231589Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5231979Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5232379Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5232809Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5233183Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5233562Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5233929Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5234306Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5234668Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5235036Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5235420Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5235804Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5236213Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5236607Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5236971Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5237411Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5237803Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5238175Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5238532Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5238901Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5239302Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5239663Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5240039Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5240439Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5240814Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5241205Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5241595Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5241975Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5242385Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5242791Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5243213Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5243670Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-08-17T13:09:19.5244811Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-08-17T13:09:19.5245796Z test_forward_backward (__main__.ReducerTest) 2022-08-17T13:09:19.5246367Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-08-17T13:09:19.5246643Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-08-17T13:09:19.5246917Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-08-17T13:09:19.5247183Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-08-17T13:09:19.5247448Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-08-17T13:09:19.5247763Z ]> 2022-08-17T13:09:19.5248086Z test_logging_init (__main__.RendezvousEnvTest) 2022-08-17T13:09:19.5248331Z 2022-08-17T13:09:19.5248638Z ]> 2022-08-17T13:09:19.5248960Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-08-17T13:09:20.4809745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:20.4810152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:20.4811467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:20.4811842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:20.6337777Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:20.6348070Z 2022-08-17T13:09:20.6348542Z Running tests... 2022-08-17T13:09:20.6348985Z ---------------------------------------------------------------------- 2022-08-17T13:09:20.8982718Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31818 2022-08-17T13:09:20.9009110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31819 2022-08-17T13:09:21.8726982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:21.8727561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:21.8728849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:21.8729222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:21.8955537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:21.8956094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:21.8958386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:21.8958960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:22.0296365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:22.0508486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:22.4049461Z ok (1.770s) 2022-08-17T13:09:22.4049719Z 2022-08-17T13:09:22.4050238Z ---------------------------------------------------------------------- 2022-08-17T13:09:22.4050680Z Ran 1 test in 1.770s 2022-08-17T13:09:22.4050809Z 2022-08-17T13:09:22.4050873Z OK 2022-08-17T13:09:22.4050956Z 2022-08-17T13:09:22.4051053Z Generating XML reports... 2022-08-17T13:09:22.4083823Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130920.xml 2022-08-17T13:09:23.6850947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:23.6851451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:23.6852635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:23.6853154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:23.8421365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:23.8431791Z 2022-08-17T13:09:23.8431930Z Running tests... 2022-08-17T13:09:23.8433092Z ---------------------------------------------------------------------- 2022-08-17T13:09:24.1142778Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31857 2022-08-17T13:09:24.1170508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31858 2022-08-17T13:09:25.0856075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:25.0856533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:25.0858143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:25.0858527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:25.0885745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:25.0886439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:25.0889007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:25.0889380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:25.2391138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:25.2424980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:25.5208940Z skip: Need at least 2 CUDA devices (1.677s) 2022-08-17T13:09:25.5209237Z 2022-08-17T13:09:25.5209548Z ---------------------------------------------------------------------- 2022-08-17T13:09:25.5209835Z Ran 1 test in 1.678s 2022-08-17T13:09:25.5210016Z 2022-08-17T13:09:25.5210092Z OK (skipped=1) 2022-08-17T13:09:25.5210204Z 2022-08-17T13:09:25.5210293Z Generating XML reports... 2022-08-17T13:09:25.5243815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130923.xml 2022-08-17T13:09:26.8329543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:26.8329959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:26.8331921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:26.8332297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:26.9872001Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:26.9882479Z 2022-08-17T13:09:26.9882819Z Running tests... 2022-08-17T13:09:26.9883306Z ---------------------------------------------------------------------- 2022-08-17T13:09:27.2512221Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31890 2022-08-17T13:09:27.2539376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31891 2022-08-17T13:09:28.2161002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:28.2161957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:28.2162797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:28.2163419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:28.2187912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:28.2188607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:28.2191425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:28.2192042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:28.3700228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:28.3733697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:28.3942830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:09:28.3943348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:09:28.3944367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:09:28.3945019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:09:28.6578012Z ok (1.669s) 2022-08-17T13:09:28.6578296Z 2022-08-17T13:09:28.6578839Z ---------------------------------------------------------------------- 2022-08-17T13:09:28.6579172Z Ran 1 test in 1.669s 2022-08-17T13:09:28.6579295Z 2022-08-17T13:09:28.6579361Z OK 2022-08-17T13:09:28.6579444Z 2022-08-17T13:09:28.6579539Z Generating XML reports... 2022-08-17T13:09:28.6612260Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130926.xml 2022-08-17T13:09:29.9683533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:29.9683983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:29.9685473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:29.9685961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:30.1268050Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:30.1278700Z 2022-08-17T13:09:30.1278815Z Running tests... 2022-08-17T13:09:30.1279450Z ---------------------------------------------------------------------- 2022-08-17T13:09:30.3929223Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31929 2022-08-17T13:09:30.3957244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31930 2022-08-17T13:09:31.3592198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:31.3592794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:31.3595588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:31.3596006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:31.3640856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:31.3641275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:31.3643946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:31.3644395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:31.5149208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:31.5190349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:31.7995756Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T13:09:31.7995948Z 2022-08-17T13:09:31.7996285Z ---------------------------------------------------------------------- 2022-08-17T13:09:31.7996548Z Ran 1 test in 1.672s 2022-08-17T13:09:31.7996670Z 2022-08-17T13:09:31.7996799Z OK (skipped=1) 2022-08-17T13:09:31.7996947Z 2022-08-17T13:09:31.7997035Z Generating XML reports... 2022-08-17T13:09:31.8030721Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130930.xml 2022-08-17T13:09:33.1227696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:33.1228418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:33.1229761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:33.1230374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:33.2799618Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:33.2810382Z 2022-08-17T13:09:33.2810652Z Running tests... 2022-08-17T13:09:33.2811280Z ---------------------------------------------------------------------- 2022-08-17T13:09:33.5466622Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31962 2022-08-17T13:09:33.5494824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31963 2022-08-17T13:09:34.5325047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:34.5325476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:34.5327306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:34.5327667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:34.5758622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:34.5758995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:34.5761038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:34.5761401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:34.6862371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:34.7290128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:35.0536521Z skip: Need at least 2 CUDA devices (1.772s) 2022-08-17T13:09:35.0536803Z 2022-08-17T13:09:35.0537268Z ---------------------------------------------------------------------- 2022-08-17T13:09:35.0537681Z Ran 1 test in 1.773s 2022-08-17T13:09:35.0537866Z 2022-08-17T13:09:35.0537993Z OK (skipped=1) 2022-08-17T13:09:35.0538179Z 2022-08-17T13:09:35.0538329Z Generating XML reports... 2022-08-17T13:09:35.0571368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130933.xml 2022-08-17T13:09:36.3511502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:36.3511893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:36.3513904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:36.3514519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:36.5066757Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:36.5077659Z 2022-08-17T13:09:36.5077761Z Running tests... 2022-08-17T13:09:36.5078529Z ---------------------------------------------------------------------- 2022-08-17T13:09:36.7727100Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31995 2022-08-17T13:09:36.7756210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31996 2022-08-17T13:09:37.7341241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:37.7341802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:37.7343091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:37.7343862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:37.7371775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:37.7372265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:37.7375511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:37.7375884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:37.8901599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:37.8923506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:38.1792486Z skip: Need at least 4 CUDA devices (1.671s) 2022-08-17T13:09:38.1792792Z 2022-08-17T13:09:38.1793181Z ---------------------------------------------------------------------- 2022-08-17T13:09:38.1793455Z Ran 1 test in 1.671s 2022-08-17T13:09:38.1793627Z 2022-08-17T13:09:38.1793711Z OK (skipped=1) 2022-08-17T13:09:38.1793829Z 2022-08-17T13:09:38.1793919Z Generating XML reports... 2022-08-17T13:09:38.1827476Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130936.xml 2022-08-17T13:09:39.5055743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:39.5056169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:39.5058448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:39.5058832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:39.6611139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:39.6621694Z 2022-08-17T13:09:39.6621859Z Running tests... 2022-08-17T13:09:39.6622355Z ---------------------------------------------------------------------- 2022-08-17T13:09:39.9303887Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32028 2022-08-17T13:09:39.9333011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32029 2022-08-17T13:09:40.9022630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:40.9023203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:40.9025205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:40.9025630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:40.9055014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:40.9055655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:40.9058635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:40.9059036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:41.0579086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:41.0609621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:41.3370867Z skip: Need at least 2 CUDA devices (1.675s) 2022-08-17T13:09:41.3371214Z 2022-08-17T13:09:41.3371714Z ---------------------------------------------------------------------- 2022-08-17T13:09:41.3371979Z Ran 1 test in 1.675s 2022-08-17T13:09:41.3372100Z 2022-08-17T13:09:41.3372164Z OK (skipped=1) 2022-08-17T13:09:41.3372278Z 2022-08-17T13:09:41.3372372Z Generating XML reports... 2022-08-17T13:09:41.3405133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130939.xml 2022-08-17T13:09:42.6601665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:42.6602592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:42.6603746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:42.6604344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:42.8112295Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:42.8122979Z 2022-08-17T13:09:42.8123382Z Running tests... 2022-08-17T13:09:42.8123819Z ---------------------------------------------------------------------- 2022-08-17T13:09:43.0732453Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32061 2022-08-17T13:09:43.0759627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32062 2022-08-17T13:09:44.0588160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:44.0588668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:44.0589826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:44.0590307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:44.0590949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:44.0591328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:44.0593412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:44.0593795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:44.2122011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:44.2138498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:44.4796678Z skip: Need at least 2 CUDA devices (1.667s) 2022-08-17T13:09:44.4796919Z 2022-08-17T13:09:44.4797242Z ---------------------------------------------------------------------- 2022-08-17T13:09:44.4797507Z Ran 1 test in 1.667s 2022-08-17T13:09:44.4797676Z 2022-08-17T13:09:44.4797756Z OK (skipped=1) 2022-08-17T13:09:44.4797873Z 2022-08-17T13:09:44.4797964Z Generating XML reports... 2022-08-17T13:09:44.4830819Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130942.xml 2022-08-17T13:09:45.7643005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:45.7643670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:45.7644981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:45.7645520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:45.9188574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:45.9198446Z 2022-08-17T13:09:45.9198632Z Running tests... 2022-08-17T13:09:45.9199239Z ---------------------------------------------------------------------- 2022-08-17T13:09:45.9205463Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-08-17T13:09:46.1815534Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32094 2022-08-17T13:09:46.1842172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32095 2022-08-17T13:09:47.1500503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:47.1501160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:47.1502826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:47.1503410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:47.1541197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:47.1541653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:47.1544931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:47.1545338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:47.3031881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:47.3069352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:47.5877264Z skip: Need at least 2 CUDA devices (1.668s) 2022-08-17T13:09:47.5877552Z 2022-08-17T13:09:47.5878057Z ---------------------------------------------------------------------- 2022-08-17T13:09:47.5878468Z Ran 1 test in 1.668s 2022-08-17T13:09:47.5878649Z 2022-08-17T13:09:47.5878754Z OK (skipped=1) 2022-08-17T13:09:47.5878942Z 2022-08-17T13:09:47.5879088Z Generating XML reports... 2022-08-17T13:09:47.5916836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130945.xml 2022-08-17T13:09:48.8981471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:48.8981978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:48.8983280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:48.8983824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:49.0536134Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:49.0546206Z 2022-08-17T13:09:49.0546348Z Running tests... 2022-08-17T13:09:49.0546981Z ---------------------------------------------------------------------- 2022-08-17T13:09:49.0553646Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:09:49.3212422Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32127 2022-08-17T13:09:49.3240713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32128 2022-08-17T13:09:50.3173266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:50.3174266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:50.3175120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:50.3175773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:50.3380996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:50.3381581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:50.3383548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:50.3384238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:50.4732594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:50.4920850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:50.8278165Z skip: Need at least 2 CUDA devices (1.773s) 2022-08-17T13:09:50.8278462Z 2022-08-17T13:09:50.8279008Z ---------------------------------------------------------------------- 2022-08-17T13:09:50.8279666Z Ran 1 test in 1.773s 2022-08-17T13:09:50.8279796Z 2022-08-17T13:09:50.8279874Z OK (skipped=1) 2022-08-17T13:09:50.8279993Z 2022-08-17T13:09:50.8280070Z Generating XML reports... 2022-08-17T13:09:50.8319352Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130949.xml 2022-08-17T13:09:52.1597289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:52.1597717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:52.1599824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:52.1600248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:52.3141250Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:52.3151390Z 2022-08-17T13:09:52.3151524Z Running tests... 2022-08-17T13:09:52.3152148Z ---------------------------------------------------------------------- 2022-08-17T13:09:52.3162093Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:52.5785854Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32160 2022-08-17T13:09:52.5814549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32161 2022-08-17T13:09:53.5552626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:53.5553147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:53.5554613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:53.5555222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:53.5607955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:53.5608446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:53.5611111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:53.5611684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:53.7081968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:53.7139544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:53.9849963Z skip: Need at least 2 CUDA devices (1.670s) 2022-08-17T13:09:53.9850233Z 2022-08-17T13:09:53.9850741Z ---------------------------------------------------------------------- 2022-08-17T13:09:53.9851118Z Ran 1 test in 1.670s 2022-08-17T13:09:53.9851239Z 2022-08-17T13:09:53.9851321Z OK (skipped=1) 2022-08-17T13:09:53.9851451Z 2022-08-17T13:09:53.9851543Z Generating XML reports... 2022-08-17T13:09:53.9887854Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130952.xml 2022-08-17T13:09:55.3110781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:55.3111305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:55.3113081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:55.3113458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:55.4665521Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:55.4676083Z 2022-08-17T13:09:55.4676210Z Running tests... 2022-08-17T13:09:55.4676806Z ---------------------------------------------------------------------- 2022-08-17T13:09:55.4685582Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:09:55.7331506Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32193 2022-08-17T13:09:55.7360012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32194 2022-08-17T13:09:56.7119163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:56.7121033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:56.7121623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:56.7122013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:56.7177919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:56.7178329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:56.7181190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:56.7181690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:56.8659909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:09:56.8724515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:09:57.1395207Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:09:57.1395553Z 2022-08-17T13:09:57.1396099Z ---------------------------------------------------------------------- 2022-08-17T13:09:57.1396439Z Ran 1 test in 1.672s 2022-08-17T13:09:57.1396561Z 2022-08-17T13:09:57.1396636Z OK (skipped=1) 2022-08-17T13:09:57.1396737Z 2022-08-17T13:09:57.1396829Z Generating XML reports... 2022-08-17T13:09:57.1433521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130955.xml 2022-08-17T13:09:58.4341472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:58.4341878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:58.4343796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:58.4344158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:58.5894116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:09:58.5904724Z 2022-08-17T13:09:58.5904820Z Running tests... 2022-08-17T13:09:58.5905260Z ---------------------------------------------------------------------- 2022-08-17T13:09:58.5912166Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:09:58.8567739Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32226 2022-08-17T13:09:58.8596770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32227 2022-08-17T13:09:59.8452398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:59.8452812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:59.8455540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:59.8455953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:59.8519959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:09:59.8520628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:09:59.8524218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:09:59.8524862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:09:59.9990072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:00.0077735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:00.2631051Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:10:00.2631364Z 2022-08-17T13:10:00.2631768Z ---------------------------------------------------------------------- 2022-08-17T13:10:00.2632047Z Ran 1 test in 1.673s 2022-08-17T13:10:00.2632153Z 2022-08-17T13:10:00.2632233Z OK (skipped=1) 2022-08-17T13:10:00.2632350Z 2022-08-17T13:10:00.2632439Z Generating XML reports... 2022-08-17T13:10:00.2668626Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130958.xml 2022-08-17T13:10:01.5668966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:01.5669443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:01.5671061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:01.5671611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:01.7260881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:01.7270876Z 2022-08-17T13:10:01.7271211Z Running tests... 2022-08-17T13:10:01.7271946Z ---------------------------------------------------------------------- 2022-08-17T13:10:01.7279008Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:10:01.9980081Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32259 2022-08-17T13:10:02.0007732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32260 2022-08-17T13:10:02.9788582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:02.9789247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:02.9790573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:02.9791175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:02.9849170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:02.9849845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:02.9851485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:02.9852134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:03.1366179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:03.1408767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:03.4043066Z skip: Need at least 2 CUDA devices (1.677s) 2022-08-17T13:10:03.4043289Z 2022-08-17T13:10:03.4043611Z ---------------------------------------------------------------------- 2022-08-17T13:10:03.4043858Z Ran 1 test in 1.677s 2022-08-17T13:10:03.4044001Z 2022-08-17T13:10:03.4044133Z OK (skipped=1) 2022-08-17T13:10:03.4044254Z 2022-08-17T13:10:03.4044345Z Generating XML reports... 2022-08-17T13:10:03.4080489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131001.xml 2022-08-17T13:10:04.7350487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:04.7351204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:04.7352291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:04.7352894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:04.8910517Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:04.8920934Z 2022-08-17T13:10:04.8921267Z Running tests... 2022-08-17T13:10:04.8921935Z ---------------------------------------------------------------------- 2022-08-17T13:10:04.8932605Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:10:05.1617597Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32292 2022-08-17T13:10:05.1648623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32293 2022-08-17T13:10:06.1510373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:06.1511021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:06.1512708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:06.1513312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:06.1551714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:06.1552340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:06.1554752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:06.1555418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:06.3085577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:06.3097740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:06.5683127Z skip: Need at least 2 CUDA devices (1.676s) 2022-08-17T13:10:06.5683429Z 2022-08-17T13:10:06.5683961Z ---------------------------------------------------------------------- 2022-08-17T13:10:06.5684249Z Ran 1 test in 1.676s 2022-08-17T13:10:06.5684371Z 2022-08-17T13:10:06.5684448Z OK (skipped=1) 2022-08-17T13:10:06.5684564Z 2022-08-17T13:10:06.5684872Z Generating XML reports... 2022-08-17T13:10:06.5724175Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131004.xml 2022-08-17T13:10:07.8968983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:07.8969390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:07.8971059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:07.8971605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:08.0534658Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:08.0545035Z 2022-08-17T13:10:08.0545152Z Running tests... 2022-08-17T13:10:08.0545499Z ---------------------------------------------------------------------- 2022-08-17T13:10:08.0555312Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:10:08.3246756Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32325 2022-08-17T13:10:08.3276631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32326 2022-08-17T13:10:09.3045396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:09.3046251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:09.3047316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:09.3047926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:09.3106212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:09.3106868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:09.3109320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:09.3109896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:09.4618071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:09.4664020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:09.7310910Z skip: Need at least 2 CUDA devices (1.676s) 2022-08-17T13:10:09.7311156Z 2022-08-17T13:10:09.7311488Z ---------------------------------------------------------------------- 2022-08-17T13:10:09.7311736Z Ran 1 test in 1.677s 2022-08-17T13:10:09.7311860Z 2022-08-17T13:10:09.7311938Z OK (skipped=1) 2022-08-17T13:10:09.7312054Z 2022-08-17T13:10:09.7312145Z Generating XML reports... 2022-08-17T13:10:09.7350130Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131008.xml 2022-08-17T13:10:11.0499960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:11.0500732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:11.0525408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:11.0525816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:11.2035623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:11.2045216Z 2022-08-17T13:10:11.2045325Z Running tests... 2022-08-17T13:10:11.2045807Z ---------------------------------------------------------------------- 2022-08-17T13:10:11.2053048Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:10:11.4730254Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32358 2022-08-17T13:10:11.4757559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32359 2022-08-17T13:10:12.4527106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:12.4527633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:12.4528690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:12.4529223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:12.4667530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:12.4668050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:12.4671151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:12.4671528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:12.6074748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:12.6197639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:12.8791593Z skip: Need at least 2 CUDA devices (1.674s) 2022-08-17T13:10:12.8791936Z 2022-08-17T13:10:12.8792365Z ---------------------------------------------------------------------- 2022-08-17T13:10:12.8792626Z Ran 1 test in 1.675s 2022-08-17T13:10:12.8792749Z 2022-08-17T13:10:12.8792811Z OK (skipped=1) 2022-08-17T13:10:12.8792927Z 2022-08-17T13:10:12.8793018Z Generating XML reports... 2022-08-17T13:10:12.8829470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131011.xml 2022-08-17T13:10:14.2004376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:14.2004774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:14.2006728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:14.2007209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:14.3564260Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:14.3574149Z 2022-08-17T13:10:14.3574286Z Running tests... 2022-08-17T13:10:14.3574780Z ---------------------------------------------------------------------- 2022-08-17T13:10:14.3585382Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:10:14.6313696Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32391 2022-08-17T13:10:14.6341249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32392 2022-08-17T13:10:15.6163045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:15.6163551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:15.6165146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:15.6165610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:15.6300043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:15.6300395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:15.6303121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:15.6303922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:15.7704900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:15.7840269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:16.0374417Z skip: Need at least 2 CUDA devices (1.680s) 2022-08-17T13:10:16.0374710Z 2022-08-17T13:10:16.0375111Z ---------------------------------------------------------------------- 2022-08-17T13:10:16.0375379Z Ran 1 test in 1.680s 2022-08-17T13:10:16.0375499Z 2022-08-17T13:10:16.0375584Z OK (skipped=1) 2022-08-17T13:10:16.0375688Z 2022-08-17T13:10:16.0375850Z Generating XML reports... 2022-08-17T13:10:16.0412974Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131014.xml 2022-08-17T13:10:17.3542122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:17.3542802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:17.3543916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:17.3544783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:17.5085431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:17.5096536Z 2022-08-17T13:10:17.5096864Z Running tests... 2022-08-17T13:10:17.5097308Z ---------------------------------------------------------------------- 2022-08-17T13:10:17.5107278Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:10:17.7797104Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32424 2022-08-17T13:10:17.7825479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32425 2022-08-17T13:10:18.7644659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:18.7645288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:18.7647251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:18.7647892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:18.7732012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:18.7732685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:18.7735160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:18.7735772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:18.9219413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:18.9282277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:19.1859233Z skip: Need at least 2 CUDA devices (1.676s) 2022-08-17T13:10:19.1859564Z 2022-08-17T13:10:19.1859979Z ---------------------------------------------------------------------- 2022-08-17T13:10:19.1860240Z Ran 1 test in 1.676s 2022-08-17T13:10:19.1860364Z 2022-08-17T13:10:19.1860440Z OK (skipped=1) 2022-08-17T13:10:19.1860554Z 2022-08-17T13:10:19.1860631Z Generating XML reports... 2022-08-17T13:10:19.1896747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131017.xml 2022-08-17T13:10:20.5082637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:20.5083305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:20.5085043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:20.5085406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:20.6641867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:20.6651610Z 2022-08-17T13:10:20.6651690Z Running tests... 2022-08-17T13:10:20.6652288Z ---------------------------------------------------------------------- 2022-08-17T13:10:20.6663115Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:10:20.9301928Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32457 2022-08-17T13:10:20.9329598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32458 2022-08-17T13:10:21.9036024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:21.9036702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:21.9038512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:21.9039100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:21.9168785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:21.9169193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:21.9171587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:21.9171993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:22.0569649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:22.0701244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:22.3364721Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T13:10:22.3365005Z 2022-08-17T13:10:22.3365489Z ---------------------------------------------------------------------- 2022-08-17T13:10:22.3366081Z Ran 1 test in 1.671s 2022-08-17T13:10:22.3366281Z 2022-08-17T13:10:22.3366399Z OK (skipped=1) 2022-08-17T13:10:22.3366572Z 2022-08-17T13:10:22.3366708Z Generating XML reports... 2022-08-17T13:10:22.3403575Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131020.xml 2022-08-17T13:10:23.6573012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:23.6573461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:23.6574809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:23.6575382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:23.8116647Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:23.8127498Z 2022-08-17T13:10:23.8127638Z Running tests... 2022-08-17T13:10:23.8128240Z ---------------------------------------------------------------------- 2022-08-17T13:10:23.8138646Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:10:24.0796805Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32490 2022-08-17T13:10:24.0826120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32491 2022-08-17T13:10:25.0446225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:25.0447268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:25.0448435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:25.0449054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:25.0827869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:25.0828409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:25.0829811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:25.0830322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:25.1962944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:25.2356504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:25.4860325Z skip: Need at least 2 CUDA devices (1.673s) 2022-08-17T13:10:25.4860641Z 2022-08-17T13:10:25.4861047Z ---------------------------------------------------------------------- 2022-08-17T13:10:25.4861316Z Ran 1 test in 1.673s 2022-08-17T13:10:25.4861437Z 2022-08-17T13:10:25.4861782Z OK (skipped=1) 2022-08-17T13:10:25.4861890Z 2022-08-17T13:10:25.4861981Z Generating XML reports... 2022-08-17T13:10:25.4898083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131023.xml 2022-08-17T13:10:26.7978610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:26.7979007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:26.7980643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:26.7981038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:26.9521609Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:26.9532147Z 2022-08-17T13:10:26.9532336Z Running tests... 2022-08-17T13:10:26.9532713Z ---------------------------------------------------------------------- 2022-08-17T13:10:26.9540393Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-08-17T13:10:27.2168131Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32523 2022-08-17T13:10:27.2196136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32524 2022-08-17T13:10:28.1940128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:28.1940579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:28.1941967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:28.1942503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:28.2181830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:28.2182334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:28.2184812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:28.2185354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:28.3475679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:28.3726453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:28.3952044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9e6kof6p 2022-08-17T13:10:28.3952924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrgvm1dd 2022-08-17T13:10:28.3953634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9e6kof6p/_remote_module_non_scriptable.py 2022-08-17T13:10:28.3954370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrgvm1dd/_remote_module_non_scriptable.py 2022-08-17T13:10:28.7233901Z ok (1.770s) 2022-08-17T13:10:28.7234158Z 2022-08-17T13:10:28.7234693Z ---------------------------------------------------------------------- 2022-08-17T13:10:28.7235042Z Ran 1 test in 1.770s 2022-08-17T13:10:28.7235167Z 2022-08-17T13:10:28.7235233Z OK 2022-08-17T13:10:28.7235329Z 2022-08-17T13:10:28.7235423Z Generating XML reports... 2022-08-17T13:10:28.7275553Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131026.xml 2022-08-17T13:10:30.0344741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:30.0345394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:30.0346403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:30.0347293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:30.1887282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:30.1897841Z 2022-08-17T13:10:30.1898135Z Running tests... 2022-08-17T13:10:30.1898799Z ---------------------------------------------------------------------- 2022-08-17T13:10:30.1906531Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-08-17T13:10:30.4554337Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32562 2022-08-17T13:10:30.4583241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32563 2022-08-17T13:10:31.4370263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:31.4371008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:31.4372114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:31.4372521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:31.4382655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:31.4383023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:31.4386560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:31.4386943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:31.5896046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:31.5905333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:31.8617317Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:10:31.8617615Z 2022-08-17T13:10:31.8618102Z ---------------------------------------------------------------------- 2022-08-17T13:10:31.8618603Z Ran 1 test in 1.672s 2022-08-17T13:10:31.8618776Z 2022-08-17T13:10:31.8618854Z OK (skipped=1) 2022-08-17T13:10:31.8618967Z 2022-08-17T13:10:31.8619058Z Generating XML reports... 2022-08-17T13:10:31.8658378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131030.xml 2022-08-17T13:10:33.1705758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:33.1706752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:33.1707937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:33.1708346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:33.3241085Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:33.3251682Z 2022-08-17T13:10:33.3251911Z Running tests... 2022-08-17T13:10:33.3252547Z ---------------------------------------------------------------------- 2022-08-17T13:10:33.3262043Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-08-17T13:10:33.5918092Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32595 2022-08-17T13:10:33.5947846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32596 2022-08-17T13:10:34.6009115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:34.6009512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:34.6011401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:34.6011790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:34.6035392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:34.6035734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:34.6038619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:34.6039243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:34.7565765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:34.7573324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:34.7845256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty4sqt__ 2022-08-17T13:10:34.7845984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57ghdi6b 2022-08-17T13:10:34.7847555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty4sqt__/_remote_module_non_scriptable.py 2022-08-17T13:10:34.7848187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57ghdi6b/_remote_module_non_scriptable.py 2022-08-17T13:10:35.0983455Z ok (1.773s) 2022-08-17T13:10:35.0983668Z 2022-08-17T13:10:35.0984011Z ---------------------------------------------------------------------- 2022-08-17T13:10:35.0984327Z Ran 1 test in 1.773s 2022-08-17T13:10:35.0984472Z 2022-08-17T13:10:35.0984549Z OK 2022-08-17T13:10:35.0984647Z 2022-08-17T13:10:35.0984731Z Generating XML reports... 2022-08-17T13:10:35.1021256Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131033.xml 2022-08-17T13:10:36.3921659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:36.3922303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:36.3923716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:36.3924330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:36.5484450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:36.5495307Z 2022-08-17T13:10:36.5495902Z Running tests... 2022-08-17T13:10:36.5496429Z ---------------------------------------------------------------------- 2022-08-17T13:10:36.5506629Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-08-17T13:10:36.8170350Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32634 2022-08-17T13:10:36.8199375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32635 2022-08-17T13:10:37.8093655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:37.8094055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:37.8095482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:37.8096000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:37.8143750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:37.8144165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:37.8146755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:37.8147373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:37.9634571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:37.9691008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:37.9869686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrn3n7wc 2022-08-17T13:10:37.9870343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuke1lzq 2022-08-17T13:10:37.9871353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrn3n7wc/_remote_module_non_scriptable.py 2022-08-17T13:10:37.9872155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuke1lzq/_remote_module_non_scriptable.py 2022-08-17T13:10:38.3235838Z ok (1.774s) 2022-08-17T13:10:38.3235994Z 2022-08-17T13:10:38.3236311Z ---------------------------------------------------------------------- 2022-08-17T13:10:38.3236581Z Ran 1 test in 1.774s 2022-08-17T13:10:38.3236706Z 2022-08-17T13:10:38.3236781Z OK 2022-08-17T13:10:38.3236882Z 2022-08-17T13:10:38.3236976Z Generating XML reports... 2022-08-17T13:10:38.3272766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131036.xml 2022-08-17T13:10:39.6185482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:39.6185883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:39.6187832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:39.6188226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:39.7735458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:39.7746038Z 2022-08-17T13:10:39.7746508Z Running tests... 2022-08-17T13:10:39.7747114Z ---------------------------------------------------------------------- 2022-08-17T13:10:39.7756003Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-08-17T13:10:40.0396462Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32679 2022-08-17T13:10:40.0424590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32680 2022-08-17T13:10:41.0115247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:41.0115898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:41.0117401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:41.0118004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:41.0121743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:41.0122131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:41.0125058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:41.0125596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:41.1671818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:41.1673863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:41.1947222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eg2q36_ 2022-08-17T13:10:41.1947910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpll01t22h 2022-08-17T13:10:41.1948954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eg2q36_/_remote_module_non_scriptable.py 2022-08-17T13:10:41.1949764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpll01t22h/_remote_module_non_scriptable.py 2022-08-17T13:10:41.5460127Z ok (1.771s) 2022-08-17T13:10:41.5460295Z 2022-08-17T13:10:41.5460703Z ---------------------------------------------------------------------- 2022-08-17T13:10:41.5461006Z Ran 1 test in 1.771s 2022-08-17T13:10:41.5461127Z 2022-08-17T13:10:41.5461194Z OK 2022-08-17T13:10:41.5461337Z 2022-08-17T13:10:41.5461463Z Generating XML reports... 2022-08-17T13:10:41.5498077Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131039.xml 2022-08-17T13:10:42.8208195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:42.8208596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:42.8210965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:42.8211386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:42.9742055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:42.9752097Z 2022-08-17T13:10:42.9752240Z Running tests... 2022-08-17T13:10:42.9752657Z ---------------------------------------------------------------------- 2022-08-17T13:10:42.9765468Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-08-17T13:10:43.2412227Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32718 2022-08-17T13:10:43.2440602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32719 2022-08-17T13:10:44.2184850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:44.2185317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:44.2186527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:44.2186984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:44.2288000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:44.2288374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:44.2290704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:44.2291079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:44.3724198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:44.3908110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:44.4097680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabyr1xgq 2022-08-17T13:10:44.4098360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_m86kmro 2022-08-17T13:10:44.4099028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabyr1xgq/_remote_module_non_scriptable.py 2022-08-17T13:10:44.4099727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_m86kmro/_remote_module_non_scriptable.py 2022-08-17T13:10:44.7476149Z ok (1.772s) 2022-08-17T13:10:44.7476412Z 2022-08-17T13:10:44.7476853Z ---------------------------------------------------------------------- 2022-08-17T13:10:44.7477138Z Ran 1 test in 1.772s 2022-08-17T13:10:44.7477274Z 2022-08-17T13:10:44.7477326Z OK 2022-08-17T13:10:44.7477422Z 2022-08-17T13:10:44.7477519Z Generating XML reports... 2022-08-17T13:10:44.7513579Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131042.xml 2022-08-17T13:10:46.0307095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:46.0307563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:46.0308446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:46.0308825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:46.1836098Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:46.1846599Z 2022-08-17T13:10:46.1846737Z Running tests... 2022-08-17T13:10:46.1847460Z ---------------------------------------------------------------------- 2022-08-17T13:10:46.1865281Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-08-17T13:10:46.4626620Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32757 2022-08-17T13:10:46.4652086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32758 2022-08-17T13:10:47.4319380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:47.4319921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:47.4321646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:47.4322319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:47.4626476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:47.4626856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:47.4629309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:47.4629692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:47.5870018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:47.6178372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:47.8688405Z skip: Need at least 2 CUDA devices (1.684s) 2022-08-17T13:10:47.8688877Z 2022-08-17T13:10:47.8689232Z ---------------------------------------------------------------------- 2022-08-17T13:10:47.8689613Z Ran 1 test in 1.684s 2022-08-17T13:10:47.8689734Z 2022-08-17T13:10:47.8689798Z OK (skipped=1) 2022-08-17T13:10:47.8690148Z 2022-08-17T13:10:47.8690241Z Generating XML reports... 2022-08-17T13:10:47.8728991Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131046.xml 2022-08-17T13:10:49.1701603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:49.1702162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:49.1703124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:49.1703612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:49.3222157Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:49.3232435Z 2022-08-17T13:10:49.3232591Z Running tests... 2022-08-17T13:10:49.3233211Z ---------------------------------------------------------------------- 2022-08-17T13:10:49.5865822Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 322 2022-08-17T13:10:49.5895119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 323 2022-08-17T13:10:50.5435427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:50.5435980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:50.5436893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:50.5437415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:50.5574028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:50.5574420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:50.5577486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:50.5577876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:50.6968957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:50.7102211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:50.9935455Z skip: Need at least 2 CUDA devices (1.670s) 2022-08-17T13:10:50.9935687Z 2022-08-17T13:10:50.9936062Z ---------------------------------------------------------------------- 2022-08-17T13:10:50.9936347Z Ran 1 test in 1.670s 2022-08-17T13:10:50.9936473Z 2022-08-17T13:10:50.9936536Z OK (skipped=1) 2022-08-17T13:10:50.9936647Z 2022-08-17T13:10:50.9936804Z Generating XML reports... 2022-08-17T13:10:50.9969605Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131049.xml 2022-08-17T13:10:52.3113161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:52.3113556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:52.3115044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:52.3115431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:52.4670238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:52.4680651Z 2022-08-17T13:10:52.4680793Z Running tests... 2022-08-17T13:10:52.4681277Z ---------------------------------------------------------------------- 2022-08-17T13:10:52.7343674Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 355 2022-08-17T13:10:52.7372303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 356 2022-08-17T13:10:53.7571735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:53.7572228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:53.7573946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:53.7574335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:53.7586828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:53.7587224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:53.7590296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:53.7590694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:53.9111756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:53.9137809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:54.2413473Z skip: Need at least 2 CUDA devices (1.773s) 2022-08-17T13:10:54.2414201Z 2022-08-17T13:10:54.2414601Z ---------------------------------------------------------------------- 2022-08-17T13:10:54.2414866Z Ran 1 test in 1.773s 2022-08-17T13:10:54.2414988Z 2022-08-17T13:10:54.2415052Z OK (skipped=1) 2022-08-17T13:10:54.2415165Z 2022-08-17T13:10:54.2415256Z Generating XML reports... 2022-08-17T13:10:54.2448406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131052.xml 2022-08-17T13:10:55.5476849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:55.5477288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:55.5479023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:55.5479403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:55.7060721Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:55.7071332Z 2022-08-17T13:10:55.7071499Z Running tests... 2022-08-17T13:10:55.7072063Z ---------------------------------------------------------------------- 2022-08-17T13:10:55.9742957Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 388 2022-08-17T13:10:55.9771384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 389 2022-08-17T13:10:56.9410772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:56.9411450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:56.9412819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:56.9413461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:56.9670452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:56.9671109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:56.9672528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:56.9673173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:57.0936838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:10:57.1223882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:10:57.3807309Z skip: Need at least 2 CUDA devices (1.673s) 2022-08-17T13:10:57.3807561Z 2022-08-17T13:10:57.3808102Z ---------------------------------------------------------------------- 2022-08-17T13:10:57.3808520Z Ran 1 test in 1.674s 2022-08-17T13:10:57.3808642Z 2022-08-17T13:10:57.3808740Z OK (skipped=1) 2022-08-17T13:10:57.3808841Z 2022-08-17T13:10:57.3808937Z Generating XML reports... 2022-08-17T13:10:57.3841729Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131055.xml 2022-08-17T13:10:58.6951607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:10:58.6951995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:10:58.6953631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:10:58.6954026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:10:58.8493291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:10:58.8502754Z 2022-08-17T13:10:58.8502839Z Running tests... 2022-08-17T13:10:58.8504010Z ---------------------------------------------------------------------- 2022-08-17T13:10:59.1137333Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 421 2022-08-17T13:10:59.1165076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 422 2022-08-17T13:11:00.0932515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:00.0932913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:00.0935543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:00.0935971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:00.0958857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:00.0959301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:00.0962025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:00.0962401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:00.2485390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:00.2491870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:00.5201685Z skip: Need at least 2 CUDA devices (1.670s) 2022-08-17T13:11:00.5201917Z 2022-08-17T13:11:00.5202318Z ---------------------------------------------------------------------- 2022-08-17T13:11:00.5202583Z Ran 1 test in 1.670s 2022-08-17T13:11:00.5202710Z 2022-08-17T13:11:00.5202779Z OK (skipped=1) 2022-08-17T13:11:00.5202893Z 2022-08-17T13:11:00.5202984Z Generating XML reports... 2022-08-17T13:11:00.5236606Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131058.xml 2022-08-17T13:11:01.8349068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:01.8349523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:01.8351170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:01.8351572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:01.9891464Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:01.9902557Z 2022-08-17T13:11:01.9902839Z Running tests... 2022-08-17T13:11:01.9903543Z ---------------------------------------------------------------------- 2022-08-17T13:11:02.2569810Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 454 2022-08-17T13:11:02.2597671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 455 2022-08-17T13:11:03.2408030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:03.2408497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:03.2410645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:03.2411027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:03.2443897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:03.2444270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:03.2447611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:03.2448082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:03.3946771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:03.3978932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:03.6634647Z skip: Need at least 2 CUDA devices (1.673s) 2022-08-17T13:11:03.6634931Z 2022-08-17T13:11:03.6635388Z ---------------------------------------------------------------------- 2022-08-17T13:11:03.6635756Z Ran 1 test in 1.673s 2022-08-17T13:11:03.6635925Z 2022-08-17T13:11:03.6636062Z OK (skipped=1) 2022-08-17T13:11:03.6636291Z 2022-08-17T13:11:03.6636463Z Generating XML reports... 2022-08-17T13:11:03.6669755Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131101.xml 2022-08-17T13:11:04.9681593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:04.9682060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:04.9683627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:04.9684085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:05.1227281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:05.1237093Z 2022-08-17T13:11:05.1237284Z Running tests... 2022-08-17T13:11:05.1237692Z ---------------------------------------------------------------------- 2022-08-17T13:11:05.3870551Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 487 2022-08-17T13:11:05.3898925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 488 2022-08-17T13:11:06.3553660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:06.3554364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:06.3555124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:06.3555684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:06.3556479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:06.3557112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:06.3558182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:06.3558767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:06.5104673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:06.5107993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:06.7934555Z skip: Need at least 4 CUDA devices (1.669s) 2022-08-17T13:11:06.7934803Z 2022-08-17T13:11:06.7935213Z ---------------------------------------------------------------------- 2022-08-17T13:11:06.7935546Z Ran 1 test in 1.670s 2022-08-17T13:11:06.7935730Z 2022-08-17T13:11:06.7935810Z OK (skipped=1) 2022-08-17T13:11:06.7935923Z 2022-08-17T13:11:06.7937025Z Generating XML reports... 2022-08-17T13:11:06.7972379Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131105.xml 2022-08-17T13:11:08.1141417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:08.1141871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:08.1143807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:08.1144180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:08.2701134Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:08.2710868Z 2022-08-17T13:11:08.2711005Z Running tests... 2022-08-17T13:11:08.2711510Z ---------------------------------------------------------------------- 2022-08-17T13:11:08.5344236Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 520 2022-08-17T13:11:08.5372018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 521 2022-08-17T13:11:09.5317901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:09.5318558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:09.5319836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:09.5320451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:09.5705956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:09.5706514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:09.5707956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:09.5708490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:09.6850687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:09.7256115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:10.0411384Z skip: Need at least 8 CUDA devices (1.770s) 2022-08-17T13:11:10.0411677Z 2022-08-17T13:11:10.0412238Z ---------------------------------------------------------------------- 2022-08-17T13:11:10.0412554Z Ran 1 test in 1.770s 2022-08-17T13:11:10.0413632Z 2022-08-17T13:11:10.0414096Z OK (skipped=1) 2022-08-17T13:11:10.0414238Z 2022-08-17T13:11:10.0414339Z Generating XML reports... 2022-08-17T13:11:10.0469907Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131108.xml 2022-08-17T13:11:11.3571506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:11.3571905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:11.3573694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:11.3574056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:11.5111426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:11.5122158Z 2022-08-17T13:11:11.5122251Z Running tests... 2022-08-17T13:11:11.5122680Z ---------------------------------------------------------------------- 2022-08-17T13:11:11.7756608Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 553 2022-08-17T13:11:11.7783520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 554 2022-08-17T13:11:12.7587200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:12.7587792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:12.7588840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:12.7589257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:12.7888913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:12.7889298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:12.7891566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:12.7892051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:12.9162240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:12.9474953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:12.9757982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce3_6ikz 2022-08-17T13:11:12.9758688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtopfo0z 2022-08-17T13:11:12.9759538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce3_6ikz/_remote_module_non_scriptable.py 2022-08-17T13:11:12.9760305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtopfo0z/_remote_module_non_scriptable.py 2022-08-17T13:11:12.9917485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:12.9918227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:13.2823071Z ok (1.769s) 2022-08-17T13:11:13.2823340Z 2022-08-17T13:11:13.2823784Z ---------------------------------------------------------------------- 2022-08-17T13:11:13.2824071Z Ran 1 test in 1.770s 2022-08-17T13:11:13.2824211Z 2022-08-17T13:11:13.2824261Z OK 2022-08-17T13:11:13.2824357Z 2022-08-17T13:11:13.2824457Z Generating XML reports... 2022-08-17T13:11:13.2856604Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131111.xml 2022-08-17T13:11:14.5629616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:14.5630009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:14.5631584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:14.5631960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:14.7160555Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:14.7170697Z 2022-08-17T13:11:14.7171128Z Running tests... 2022-08-17T13:11:14.7171554Z ---------------------------------------------------------------------- 2022-08-17T13:11:14.9832160Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 592 2022-08-17T13:11:14.9858500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 593 2022-08-17T13:11:15.9807078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:15.9807473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:15.9808986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:15.9809499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:15.9869260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:15.9869639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:15.9871837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:15.9872245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:16.1373944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:16.1404375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:16.1654953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun8e4o3m 2022-08-17T13:11:16.1655454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9knr0hoy 2022-08-17T13:11:16.1656647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun8e4o3m/_remote_module_non_scriptable.py 2022-08-17T13:11:16.1657405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9knr0hoy/_remote_module_non_scriptable.py 2022-08-17T13:11:16.1816714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:16.1817124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:16.4897968Z ok (1.772s) 2022-08-17T13:11:16.4898231Z 2022-08-17T13:11:16.4898730Z ---------------------------------------------------------------------- 2022-08-17T13:11:16.4898999Z Ran 1 test in 1.773s 2022-08-17T13:11:16.4899108Z 2022-08-17T13:11:16.4899172Z OK 2022-08-17T13:11:16.4899266Z 2022-08-17T13:11:16.4899362Z Generating XML reports... 2022-08-17T13:11:16.4932040Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131114.xml 2022-08-17T13:11:17.7863007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:17.7863557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:17.7864365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:17.7864777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:17.9422415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:17.9432465Z 2022-08-17T13:11:17.9432816Z Running tests... 2022-08-17T13:11:17.9433304Z ---------------------------------------------------------------------- 2022-08-17T13:11:17.9449027Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-08-17T13:11:18.2096415Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 631 2022-08-17T13:11:18.2125790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 632 2022-08-17T13:11:19.1733080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:19.1733805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:19.1735434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:19.1735829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:19.1743597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:19.1743962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:19.1746843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:19.1747221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:19.3266093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:19.3271988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:19.3552396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk988dmvs 2022-08-17T13:11:19.3553047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvrqu4oa 2022-08-17T13:11:19.3553885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk988dmvs/_remote_module_non_scriptable.py 2022-08-17T13:11:19.3554568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvrqu4oa/_remote_module_non_scriptable.py 2022-08-17T13:11:19.3698719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:19.3702995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:11:19.7162059Z ok (1.773s) 2022-08-17T13:11:19.7162303Z 2022-08-17T13:11:19.7162682Z ---------------------------------------------------------------------- 2022-08-17T13:11:19.7162962Z Ran 1 test in 1.773s 2022-08-17T13:11:19.7163135Z 2022-08-17T13:11:19.7163204Z OK 2022-08-17T13:11:19.7163300Z 2022-08-17T13:11:19.7163396Z Generating XML reports... 2022-08-17T13:11:19.7199399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131117.xml 2022-08-17T13:11:21.0427971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:21.0428611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:21.0429447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:21.0430035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:21.1969362Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:21.1979632Z 2022-08-17T13:11:21.1979932Z Running tests... 2022-08-17T13:11:21.1980583Z ---------------------------------------------------------------------- 2022-08-17T13:11:21.1997016Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-08-17T13:11:21.4605812Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 677 2022-08-17T13:11:21.4633631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 678 2022-08-17T13:11:22.4228479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:22.4228948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:22.4230923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:22.4231438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:22.4255435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:22.4256171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:22.4258501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:22.4258916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:22.5774683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:22.5795331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:22.6074943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx5bj011 2022-08-17T13:11:22.6075612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zcp50a9 2022-08-17T13:11:22.6076302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx5bj011/_remote_module_non_scriptable.py 2022-08-17T13:11:22.6077020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zcp50a9/_remote_module_non_scriptable.py 2022-08-17T13:11:22.9669696Z ok (1.769s) 2022-08-17T13:11:22.9669948Z 2022-08-17T13:11:22.9670421Z ---------------------------------------------------------------------- 2022-08-17T13:11:22.9670846Z Ran 1 test in 1.769s 2022-08-17T13:11:22.9671344Z 2022-08-17T13:11:22.9671448Z OK 2022-08-17T13:11:22.9671580Z 2022-08-17T13:11:22.9671731Z Generating XML reports... 2022-08-17T13:11:22.9708570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131121.xml 2022-08-17T13:11:24.2910651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:24.2911028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:24.2913390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:24.2913790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:24.4448430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:24.4458726Z 2022-08-17T13:11:24.4458804Z Running tests... 2022-08-17T13:11:24.4459271Z ---------------------------------------------------------------------- 2022-08-17T13:11:24.7122577Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 722 2022-08-17T13:11:24.7152285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 723 2022-08-17T13:11:25.7030601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:25.7031317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:25.7031787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:25.7032174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:25.7122573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:25.7123062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:25.7125423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:25.7126158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:25.8580486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:25.8585398Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8587124Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8588114Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8588982Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8589925Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8590785Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8730427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:25.8736930Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8738468Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8739479Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8740392Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8741244Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:25.8742097Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:11:26.1192465Z ok (1.673s) 2022-08-17T13:11:26.1192691Z 2022-08-17T13:11:26.1193243Z ---------------------------------------------------------------------- 2022-08-17T13:11:26.1193652Z Ran 1 test in 1.673s 2022-08-17T13:11:26.1193815Z 2022-08-17T13:11:26.1193878Z OK 2022-08-17T13:11:26.1193975Z 2022-08-17T13:11:26.1194074Z Generating XML reports... 2022-08-17T13:11:26.1227326Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131124.xml 2022-08-17T13:11:27.4256718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:27.4257173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:27.4258337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:27.4258731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:27.5801180Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:27.5810822Z 2022-08-17T13:11:27.5811203Z Running tests... 2022-08-17T13:11:27.5811705Z ---------------------------------------------------------------------- 2022-08-17T13:11:27.8469445Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 755 2022-08-17T13:11:27.8497897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 756 2022-08-17T13:11:28.8285326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:28.8285757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:28.8286604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:28.8287149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:28.8287996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:28.8288645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:28.8289193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:28.8289558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:28.9835776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:28.9839675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:29.2535467Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:11:29.2535828Z 2022-08-17T13:11:29.2536231Z ---------------------------------------------------------------------- 2022-08-17T13:11:29.2536495Z Ran 1 test in 1.672s 2022-08-17T13:11:29.2536615Z 2022-08-17T13:11:29.2536691Z OK (skipped=1) 2022-08-17T13:11:29.2536808Z 2022-08-17T13:11:29.2536896Z Generating XML reports... 2022-08-17T13:11:29.2572554Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131127.xml 2022-08-17T13:11:30.5681605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:30.5682073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:30.5683927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:30.5684287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:30.7259853Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:30.7270073Z 2022-08-17T13:11:30.7270162Z Running tests... 2022-08-17T13:11:30.7271090Z ---------------------------------------------------------------------- 2022-08-17T13:11:30.9905541Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 788 2022-08-17T13:11:30.9932954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 789 2022-08-17T13:11:31.9591618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:31.9592013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:31.9594989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:31.9595634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:32.0076116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:32.0076703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:32.0078493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:32.0079005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:32.1118305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:32.1592701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:32.1873055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7hl_gru 2022-08-17T13:11:32.1873758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6ltdebr 2022-08-17T13:11:32.1874720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7hl_gru/_remote_module_non_scriptable.py 2022-08-17T13:11:32.1875639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6ltdebr/_remote_module_non_scriptable.py 2022-08-17T13:11:32.4972653Z ok (1.770s) 2022-08-17T13:11:32.4972941Z 2022-08-17T13:11:32.4973455Z ---------------------------------------------------------------------- 2022-08-17T13:11:32.4973706Z Ran 1 test in 1.770s 2022-08-17T13:11:32.4973829Z 2022-08-17T13:11:32.4973892Z OK 2022-08-17T13:11:32.4973986Z 2022-08-17T13:11:32.4974081Z Generating XML reports... 2022-08-17T13:11:32.5007060Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131130.xml 2022-08-17T13:11:33.7885703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:33.7886339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:33.7887573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:33.7887969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:33.9437371Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:33.9447722Z 2022-08-17T13:11:33.9447812Z Running tests... 2022-08-17T13:11:33.9448408Z ---------------------------------------------------------------------- 2022-08-17T13:11:34.2116613Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 833 2022-08-17T13:11:34.2143255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 834 2022-08-17T13:11:35.2005199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:35.2005725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:35.2006858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:35.2007329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:35.2030101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:35.2030718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:35.2033312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:35.2033982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:35.3580007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:35.3582851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:35.3861783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeirag2x9 2022-08-17T13:11:35.3862364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uhuyb3w 2022-08-17T13:11:35.3863686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeirag2x9/_remote_module_non_scriptable.py 2022-08-17T13:11:35.3864330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uhuyb3w/_remote_module_non_scriptable.py 2022-08-17T13:11:35.7183040Z ok (1.773s) 2022-08-17T13:11:35.7183285Z 2022-08-17T13:11:35.7183813Z ---------------------------------------------------------------------- 2022-08-17T13:11:35.7184067Z Ran 1 test in 1.773s 2022-08-17T13:11:35.7184189Z 2022-08-17T13:11:35.7184254Z OK 2022-08-17T13:11:35.7184351Z 2022-08-17T13:11:35.7184449Z Generating XML reports... 2022-08-17T13:11:35.7217227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131133.xml 2022-08-17T13:11:37.0289858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:37.0290259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:37.0292079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:37.0292465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:37.1842075Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:37.1853558Z 2022-08-17T13:11:37.1853666Z Running tests... 2022-08-17T13:11:37.1854279Z ---------------------------------------------------------------------- 2022-08-17T13:11:37.4504241Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 878 2022-08-17T13:11:37.4531439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 879 2022-08-17T13:11:38.4418715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:38.4419302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:38.4420590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:38.4421234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:38.4427064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:38.4427710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:38.4431326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:38.4431941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:38.5957023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:38.5968578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:38.8569448Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T13:11:38.8570826Z 2022-08-17T13:11:38.8571171Z ---------------------------------------------------------------------- 2022-08-17T13:11:38.8571442Z Ran 1 test in 1.672s 2022-08-17T13:11:38.8571563Z 2022-08-17T13:11:38.8571641Z OK (skipped=1) 2022-08-17T13:11:38.8571755Z 2022-08-17T13:11:38.8571831Z Generating XML reports... 2022-08-17T13:11:38.8604003Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131137.xml 2022-08-17T13:11:40.1738106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:40.1738509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:40.1739592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:40.1740019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:40.3316604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:40.3327524Z 2022-08-17T13:11:40.3327652Z Running tests... 2022-08-17T13:11:40.3327992Z ---------------------------------------------------------------------- 2022-08-17T13:11:40.6041259Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 911 2022-08-17T13:11:40.6070559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 912 2022-08-17T13:11:41.5775823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:41.5776208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:41.5778710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:41.5779135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:41.5807533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:41.5808171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:41.5811065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:41.5811510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:41.7324775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:41.7358570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:42.0110923Z skip: Need at least 2 CUDA devices (1.678s) 2022-08-17T13:11:42.0111404Z 2022-08-17T13:11:42.0111913Z ---------------------------------------------------------------------- 2022-08-17T13:11:42.0112375Z Ran 1 test in 1.678s 2022-08-17T13:11:42.0112546Z 2022-08-17T13:11:42.0112621Z OK (skipped=1) 2022-08-17T13:11:42.0112731Z 2022-08-17T13:11:42.0112839Z Generating XML reports... 2022-08-17T13:11:42.0145883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131140.xml 2022-08-17T13:11:43.3086471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:43.3086951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:43.3088935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:43.3089467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:43.4616190Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:43.4625407Z 2022-08-17T13:11:43.4625508Z Running tests... 2022-08-17T13:11:43.4626055Z ---------------------------------------------------------------------- 2022-08-17T13:11:43.7225969Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 944 2022-08-17T13:11:43.7254306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 945 2022-08-17T13:11:43.7281419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 946 2022-08-17T13:11:43.7312187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 947 2022-08-17T13:11:44.7447913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:44.7448582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:44.7449654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:44.7450280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:44.7529854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:44.7530542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:44.7532862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:44.7533487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:44.7803846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:44.7804517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:44.7806339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:44.7806951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:44.7811576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:44.7812183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:44.7815320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:44.7816006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:44.9014881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:11:44.9087813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:44.9381308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:44.9388281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:11:45.3360424Z ok (1.873s) 2022-08-17T13:11:45.3360634Z 2022-08-17T13:11:45.3361174Z ---------------------------------------------------------------------- 2022-08-17T13:11:45.3361604Z Ran 1 test in 1.873s 2022-08-17T13:11:45.3361712Z 2022-08-17T13:11:45.3361776Z OK 2022-08-17T13:11:45.3361870Z 2022-08-17T13:11:45.3361965Z Generating XML reports... 2022-08-17T13:11:45.3395398Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131143.xml 2022-08-17T13:11:46.6590380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:46.6590988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:46.6592593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:46.6593479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:46.8129447Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:46.8139934Z 2022-08-17T13:11:46.8140386Z Running tests... 2022-08-17T13:11:46.8140823Z ---------------------------------------------------------------------- 2022-08-17T13:11:47.0763062Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1011 2022-08-17T13:11:47.0790325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1012 2022-08-17T13:11:47.0818607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1013 2022-08-17T13:11:47.0850204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1014 2022-08-17T13:11:48.1718182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:48.1718907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:48.1720502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:48.1732487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:48.1815842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:48.1816514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:48.1818974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:48.1819695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:48.1823488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:48.1824243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:48.1827896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:48.1828496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:48.2608512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:48.2609160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:48.2610516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:48.2611135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:48.3329781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:48.3409012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:11:48.3426860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:48.4384215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:11:48.6897885Z skip: Need at least 2 CUDA devices (1.876s) 2022-08-17T13:11:48.6898184Z 2022-08-17T13:11:48.6898713Z ---------------------------------------------------------------------- 2022-08-17T13:11:48.6899040Z Ran 1 test in 1.876s 2022-08-17T13:11:48.6899160Z 2022-08-17T13:11:48.6899237Z OK (skipped=1) 2022-08-17T13:11:48.6899351Z 2022-08-17T13:11:48.6899442Z Generating XML reports... 2022-08-17T13:11:48.6933433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131146.xml 2022-08-17T13:11:50.0004076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:50.0004786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:50.0005434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:50.0005810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:50.1558042Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:50.1567889Z 2022-08-17T13:11:50.1567996Z Running tests... 2022-08-17T13:11:50.1568677Z ---------------------------------------------------------------------- 2022-08-17T13:11:50.4207706Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1066 2022-08-17T13:11:50.4240703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1067 2022-08-17T13:11:50.4273046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1068 2022-08-17T13:11:50.4300935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1069 2022-08-17T13:11:51.4513208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:51.4513856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:51.4519249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:51.4519957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:51.4520761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:51.4521387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:51.4522222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:51.4522835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:51.4523699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:51.4524314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:51.4525293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:51.4526084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:51.4694149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:51.4694787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:51.4696589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:51.4697137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:51.6103889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:11:51.6131182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:51.6226753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:11:51.6305637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:52.0347613Z ok (1.878s) 2022-08-17T13:11:52.0347836Z 2022-08-17T13:11:52.0348257Z ---------------------------------------------------------------------- 2022-08-17T13:11:52.0348551Z Ran 1 test in 1.878s 2022-08-17T13:11:52.0348718Z 2022-08-17T13:11:52.0348789Z OK 2022-08-17T13:11:52.0348872Z 2022-08-17T13:11:52.0348969Z Generating XML reports... 2022-08-17T13:11:52.0383013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131150.xml 2022-08-17T13:11:53.3286078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:53.3286749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:53.3287827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:53.3288383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:53.4791582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:53.4800686Z 2022-08-17T13:11:53.4800827Z Running tests... 2022-08-17T13:11:53.4801412Z ---------------------------------------------------------------------- 2022-08-17T13:11:53.7422163Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1133 2022-08-17T13:11:53.7450635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1134 2022-08-17T13:11:53.7478420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1135 2022-08-17T13:11:53.7509303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1136 2022-08-17T13:11:54.7705615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:54.7706557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:54.7707601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:54.7708226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:54.7709220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:54.7709818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:54.7713480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:54.7714127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:54.8067532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:54.8068193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:54.8069180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:54.8069787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:54.8131416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:54.8131977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:54.8134338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:54.8134883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:54.9292553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:54.9293236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:54.9638615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:11:54.9688237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:11:54.9898470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:11:54.9999887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:11:55.0102231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:11:55.0102832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:11:55.0104003Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:11:55.0104579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:11:55.0105115Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:11:55.0203655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:11:55.3556588Z ok (1.875s) 2022-08-17T13:11:55.3556832Z 2022-08-17T13:11:55.3557215Z ---------------------------------------------------------------------- 2022-08-17T13:11:55.3557479Z Ran 1 test in 1.876s 2022-08-17T13:11:55.3557638Z 2022-08-17T13:11:55.3557701Z OK 2022-08-17T13:11:55.3557796Z 2022-08-17T13:11:55.3557907Z Generating XML reports... 2022-08-17T13:11:55.3591666Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131153.xml 2022-08-17T13:11:56.6339044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:56.6340014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:56.6340896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:56.6341582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:56.7876062Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:11:56.7886597Z 2022-08-17T13:11:56.7886879Z Running tests... 2022-08-17T13:11:56.7887559Z ---------------------------------------------------------------------- 2022-08-17T13:11:57.0516978Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1200 2022-08-17T13:11:57.0545451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1201 2022-08-17T13:11:57.0572337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1202 2022-08-17T13:11:57.0602904Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1203 2022-08-17T13:11:58.0518150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:58.0518781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:58.0519921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:58.0520571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:58.0567609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:58.0568262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:58.0570929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:58.0571548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:58.0726529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:58.0727220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:58.0729190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:58.0729821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:58.1077399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:58.1078425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:58.1079702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:58.1080266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:11:58.2109834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:11:58.2155393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:11:58.2302542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:11:58.2654559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:11:58.6651091Z ok (1.876s) 2022-08-17T13:11:58.6651321Z 2022-08-17T13:11:58.6651811Z ---------------------------------------------------------------------- 2022-08-17T13:11:58.6652243Z Ran 1 test in 1.876s 2022-08-17T13:11:58.6652451Z 2022-08-17T13:11:58.6652549Z OK 2022-08-17T13:11:58.6652709Z 2022-08-17T13:11:58.6652834Z Generating XML reports... 2022-08-17T13:11:58.6686605Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131156.xml 2022-08-17T13:11:59.9715421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:11:59.9715881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:11:59.9716670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:11:59.9717032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:00.1269494Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:00.1279545Z 2022-08-17T13:12:00.1279621Z Running tests... 2022-08-17T13:12:00.1280051Z ---------------------------------------------------------------------- 2022-08-17T13:12:00.3920815Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1267 2022-08-17T13:12:00.3949698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1268 2022-08-17T13:12:00.3977555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1269 2022-08-17T13:12:00.4009458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1270 2022-08-17T13:12:01.4018308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:01.4019004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:01.4020282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:01.4020951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:01.4206466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:01.4207114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:01.4208625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:01.4209280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:01.4590433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:01.4591085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:01.4592427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:01.4593030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:01.4701563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:01.4702328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:01.4704039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:01.4704775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:01.5625725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:01.5787138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:01.6161349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:01.6268357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:02.0057581Z ok (1.877s) 2022-08-17T13:12:02.0057916Z 2022-08-17T13:12:02.0058411Z ---------------------------------------------------------------------- 2022-08-17T13:12:02.0058679Z Ran 1 test in 1.878s 2022-08-17T13:12:02.0058804Z 2022-08-17T13:12:02.0058870Z OK 2022-08-17T13:12:02.0058955Z 2022-08-17T13:12:02.0059053Z Generating XML reports... 2022-08-17T13:12:02.0092239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131200.xml 2022-08-17T13:12:03.2948704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:03.2949417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:03.2950563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:03.2951189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:03.4517345Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:03.4527023Z 2022-08-17T13:12:03.4527503Z Running tests... 2022-08-17T13:12:03.4527929Z ---------------------------------------------------------------------- 2022-08-17T13:12:03.7164081Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1334 2022-08-17T13:12:03.7191876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1335 2022-08-17T13:12:03.7218467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1336 2022-08-17T13:12:03.7248431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1337 2022-08-17T13:12:04.7283417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:04.7284069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:04.7285721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:04.7286500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:04.7785955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:04.7786532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:04.7788088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:04.7788701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:04.8004934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:04.8005585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:04.8007268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:04.8008225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:04.8166612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:04.8167378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:04.8168593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:04.8169161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:04.8866401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:04.9370302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:04.9619174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:04.9741494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:06.7319798Z ok (3.279s) 2022-08-17T13:12:06.7320046Z 2022-08-17T13:12:06.7320528Z ---------------------------------------------------------------------- 2022-08-17T13:12:06.7320795Z Ran 1 test in 3.279s 2022-08-17T13:12:06.7320918Z 2022-08-17T13:12:06.7320987Z OK 2022-08-17T13:12:06.7321331Z 2022-08-17T13:12:06.7321416Z Generating XML reports... 2022-08-17T13:12:06.7357261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131203.xml 2022-08-17T13:12:08.0400630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:08.0401207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:08.0402413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:08.0402928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:08.1961168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:08.1971460Z 2022-08-17T13:12:08.1971623Z Running tests... 2022-08-17T13:12:08.1972254Z ---------------------------------------------------------------------- 2022-08-17T13:12:08.4593810Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1425 2022-08-17T13:12:08.4621485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1426 2022-08-17T13:12:08.4649588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1427 2022-08-17T13:12:08.4679892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1428 2022-08-17T13:12:09.5218594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:09.5219208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:09.5220048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:09.5220654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:09.5261808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:09.5262607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:09.5264216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:09.5264899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:09.5915637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:09.5916377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:09.5917625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:09.5918366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:09.6023504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:09.6025063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:09.6025871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:09.6026463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:09.6801800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:09.6852873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:09.7505721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:09.7659746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:10.0729218Z skip: Need at least 2 CUDA devices (1.875s) 2022-08-17T13:12:10.0729598Z 2022-08-17T13:12:10.0730727Z ---------------------------------------------------------------------- 2022-08-17T13:12:10.0731038Z Ran 1 test in 1.876s 2022-08-17T13:12:10.0731172Z 2022-08-17T13:12:10.0731240Z OK (skipped=1) 2022-08-17T13:12:10.0731359Z 2022-08-17T13:12:10.0731450Z Generating XML reports... 2022-08-17T13:12:10.0764094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131208.xml 2022-08-17T13:12:11.4019066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:11.4019692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:11.4042860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:11.4043287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:11.5593196Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:11.5603275Z 2022-08-17T13:12:11.5603451Z Running tests... 2022-08-17T13:12:11.5603898Z ---------------------------------------------------------------------- 2022-08-17T13:12:11.8254230Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1480 2022-08-17T13:12:11.8284615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1481 2022-08-17T13:12:11.8313405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1482 2022-08-17T13:12:11.8345071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1483 2022-08-17T13:12:12.9223830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:12.9224442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:12.9225499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:12.9226109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:12.9671040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:12.9671732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:12.9673894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:12.9674564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:12.9679344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:12.9680327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:12.9682254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:12.9682784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:12.9736118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:12.9736688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:12.9739240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:12.9739782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:13.0797787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:13.1270136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:13.1278506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:13.1325318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:13.5394919Z ok (1.979s) 2022-08-17T13:12:13.5395114Z 2022-08-17T13:12:13.5395454Z ---------------------------------------------------------------------- 2022-08-17T13:12:13.5395717Z Ran 1 test in 1.979s 2022-08-17T13:12:13.5395839Z 2022-08-17T13:12:13.5395914Z OK 2022-08-17T13:12:13.5396049Z 2022-08-17T13:12:13.5396167Z Generating XML reports... 2022-08-17T13:12:13.5430033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131211.xml 2022-08-17T13:12:14.8235778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:14.8236347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:14.8237251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:14.8237628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:14.9791103Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:14.9801018Z 2022-08-17T13:12:14.9801184Z Running tests... 2022-08-17T13:12:14.9801545Z ---------------------------------------------------------------------- 2022-08-17T13:12:15.2415353Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1547 2022-08-17T13:12:15.2442851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1548 2022-08-17T13:12:15.2471084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1549 2022-08-17T13:12:15.2500894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1550 2022-08-17T13:12:16.3242412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:16.3243034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:16.3246689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:16.3247356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:16.3643994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:16.3644545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:16.3646047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:16.3646625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:16.3788980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:16.3789542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:16.3791307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:16.3791843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:16.4049889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:16.4050551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:16.4051340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:16.4051955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:16.5030575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:16.5256365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:16.5435826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:16.5619876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:16.8548100Z skip: Need at least 2 CUDA devices (1.874s) 2022-08-17T13:12:16.8548523Z 2022-08-17T13:12:16.8549312Z ---------------------------------------------------------------------- 2022-08-17T13:12:16.8549643Z Ran 1 test in 1.875s 2022-08-17T13:12:16.8549827Z 2022-08-17T13:12:16.8549912Z OK (skipped=1) 2022-08-17T13:12:16.8550027Z 2022-08-17T13:12:16.8550125Z Generating XML reports... 2022-08-17T13:12:16.8582579Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131214.xml 2022-08-17T13:12:18.1652677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:18.1653120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:18.1655029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:18.1655409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:18.3181293Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:18.3191114Z 2022-08-17T13:12:18.3191327Z Running tests... 2022-08-17T13:12:18.3191933Z ---------------------------------------------------------------------- 2022-08-17T13:12:18.5838249Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1602 2022-08-17T13:12:18.5864932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1603 2022-08-17T13:12:18.5892988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1604 2022-08-17T13:12:18.5923043Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1605 2022-08-17T13:12:19.6158663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:19.6159327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:19.6160145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:19.6160761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:19.6374920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:19.6375587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:19.6377134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:19.6377722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:19.6600093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:19.6600751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:19.6602410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:19.6603001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:19.6896632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:19.6897830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:19.6898540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:19.6899079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:19.7755796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:19.8050622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:19.8203514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:19.8501096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:20.1970305Z skip: Need at least 2 CUDA devices (1.877s) 2022-08-17T13:12:20.1970528Z 2022-08-17T13:12:20.1970870Z ---------------------------------------------------------------------- 2022-08-17T13:12:20.1971119Z Ran 1 test in 1.878s 2022-08-17T13:12:20.1971239Z 2022-08-17T13:12:20.1971315Z OK (skipped=1) 2022-08-17T13:12:20.1971428Z 2022-08-17T13:12:20.1971530Z Generating XML reports... 2022-08-17T13:12:20.2004516Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131218.xml 2022-08-17T13:12:21.5034714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:21.5035128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:21.5037038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:21.5037401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:21.6733149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:21.6742995Z 2022-08-17T13:12:21.6743097Z Running tests... 2022-08-17T13:12:21.6743657Z ---------------------------------------------------------------------- 2022-08-17T13:12:21.9413241Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1657 2022-08-17T13:12:21.9442751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1658 2022-08-17T13:12:21.9471290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1659 2022-08-17T13:12:21.9502201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1660 2022-08-17T13:12:23.0545133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:23.0545804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:23.0547449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:23.0548090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:23.0971288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:23.0972327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:23.0973453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:23.0974108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:23.1170300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:23.1172448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:23.1173219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:23.1173806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:23.1174603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:23.1175245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:23.1176332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:23.1177225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:23.2121183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:23.2569460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:23.2748580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:23.2755113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:23.6550509Z ok (1.980s) 2022-08-17T13:12:23.6550778Z 2022-08-17T13:12:23.6551256Z ---------------------------------------------------------------------- 2022-08-17T13:12:23.6551667Z Ran 1 test in 1.981s 2022-08-17T13:12:23.6551856Z 2022-08-17T13:12:23.6551952Z OK 2022-08-17T13:12:23.6552107Z 2022-08-17T13:12:23.6552261Z Generating XML reports... 2022-08-17T13:12:23.6586522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131221.xml 2022-08-17T13:12:24.9470031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:24.9470742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:24.9472115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:24.9472711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:25.1015868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:25.1026118Z 2022-08-17T13:12:25.1026440Z Running tests... 2022-08-17T13:12:25.1027112Z ---------------------------------------------------------------------- 2022-08-17T13:12:25.3630400Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1724 2022-08-17T13:12:25.3659245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1725 2022-08-17T13:12:25.3686949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1726 2022-08-17T13:12:25.3718345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1727 2022-08-17T13:12:26.4353556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:26.4354167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:26.4355242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:26.4355900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:26.4402626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:26.4403251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:26.4405327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:26.4406153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:26.4974855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:26.4975521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:26.4976735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:26.4977308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:26.5175429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:26.5176068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:26.5177771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:26.5178409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:26.5960070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:26.5989609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:26.6561637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:26.6741589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:27.0768689Z ok (1.974s) 2022-08-17T13:12:27.0768939Z 2022-08-17T13:12:27.0769466Z ---------------------------------------------------------------------- 2022-08-17T13:12:27.0769931Z Ran 1 test in 1.974s 2022-08-17T13:12:27.0770126Z 2022-08-17T13:12:27.0770236Z OK 2022-08-17T13:12:27.0770407Z 2022-08-17T13:12:27.0770572Z Generating XML reports... 2022-08-17T13:12:27.0803350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131225.xml 2022-08-17T13:12:28.4160929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:28.4161335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:28.4163119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:28.4163482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:28.5715902Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:28.5725543Z 2022-08-17T13:12:28.5725674Z Running tests... 2022-08-17T13:12:28.5726254Z ---------------------------------------------------------------------- 2022-08-17T13:12:28.8381094Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1791 2022-08-17T13:12:28.8409077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1792 2022-08-17T13:12:28.8438241Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1793 2022-08-17T13:12:28.8469916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1794 2022-08-17T13:12:29.9239026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:29.9239685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:29.9240823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:29.9241766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:29.9531131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:29.9532717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:29.9533510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:29.9534111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:29.9534906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:29.9535530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:29.9536311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:29.9536934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:29.9625299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:29.9626244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:29.9627913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:29.9628558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:30.0802851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:30.1145079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:30.1162734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:30.1377114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:30.1572749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:12:30.1614901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:12:30.1615854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:12:30.1616636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:12:30.1617285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:12:30.1617847Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:12:30.1673950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:12:30.1717225Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:12:30.5518541Z ok (1.979s) 2022-08-17T13:12:30.5518753Z 2022-08-17T13:12:30.5519228Z ---------------------------------------------------------------------- 2022-08-17T13:12:30.5519517Z Ran 1 test in 1.979s 2022-08-17T13:12:30.5519641Z 2022-08-17T13:12:30.5519705Z OK 2022-08-17T13:12:30.5519802Z 2022-08-17T13:12:30.5519883Z Generating XML reports... 2022-08-17T13:12:30.5553066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131228.xml 2022-08-17T13:12:31.8331819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:31.8332309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:31.8333315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:31.8333936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:31.9862204Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:31.9872326Z 2022-08-17T13:12:31.9872430Z Running tests... 2022-08-17T13:12:31.9872875Z ---------------------------------------------------------------------- 2022-08-17T13:12:32.2530656Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1858 2022-08-17T13:12:32.2558206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1859 2022-08-17T13:12:32.2585672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1860 2022-08-17T13:12:32.2615803Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1861 2022-08-17T13:12:33.3788671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:33.3789410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:33.3790481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:33.3791401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:33.3803208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:33.3803963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:33.3805477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:33.3806388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:33.3867683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:33.3868432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:33.3869178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:33.3869771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:33.3870723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:33.3871370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:33.3872140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:33.3872717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:33.5398583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:33.5399525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:33.5452486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:33.5474070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:33.9666171Z ok (1.979s) 2022-08-17T13:12:33.9666517Z 2022-08-17T13:12:33.9666970Z ---------------------------------------------------------------------- 2022-08-17T13:12:33.9667225Z Ran 1 test in 1.979s 2022-08-17T13:12:33.9667345Z 2022-08-17T13:12:33.9667409Z OK 2022-08-17T13:12:33.9667505Z 2022-08-17T13:12:33.9667600Z Generating XML reports... 2022-08-17T13:12:33.9703206Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131231.xml 2022-08-17T13:12:35.2841087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:35.2841501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:35.2842272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:35.2842709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:35.4388922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:35.4399238Z 2022-08-17T13:12:35.4399387Z Running tests... 2022-08-17T13:12:35.4400029Z ---------------------------------------------------------------------- 2022-08-17T13:12:35.7067476Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1925 2022-08-17T13:12:35.7095397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1926 2022-08-17T13:12:35.7123161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1927 2022-08-17T13:12:35.7154591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1928 2022-08-17T13:12:36.7356696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:36.7357411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:36.7360593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:36.7361279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:36.7690244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:36.7690966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:36.7693805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:36.7694536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:36.7727754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:36.7728457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:36.7731663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:36.7732285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:36.8147945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:36.8148642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:36.8150293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:36.8150861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:36.8926255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:36.9287729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:36.9301093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:36.9713102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:37.4203846Z ok (1.980s) 2022-08-17T13:12:37.4204104Z 2022-08-17T13:12:37.4204567Z ---------------------------------------------------------------------- 2022-08-17T13:12:37.4204851Z Ran 1 test in 1.980s 2022-08-17T13:12:37.4204976Z 2022-08-17T13:12:37.4205041Z OK 2022-08-17T13:12:37.4205138Z 2022-08-17T13:12:37.4205221Z Generating XML reports... 2022-08-17T13:12:37.4238584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131235.xml 2022-08-17T13:12:38.7255351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:38.7256303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:38.7257153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:38.7257764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:38.8785799Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:38.8796181Z 2022-08-17T13:12:38.8796678Z Running tests... 2022-08-17T13:12:38.8797122Z ---------------------------------------------------------------------- 2022-08-17T13:12:39.1437042Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1992 2022-08-17T13:12:39.1465857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1993 2022-08-17T13:12:39.1495582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1994 2022-08-17T13:12:39.1526431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1995 2022-08-17T13:12:40.1682657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:40.1683667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:40.1684664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:40.1685349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:40.1690178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:40.1690830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:40.1693633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:40.1694340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:40.1941774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:40.1943238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:40.1944175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:40.1944918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:40.2340101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:40.2341451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:40.2342433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:40.2343101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:40.3286570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:40.3296567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:40.3523741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:40.4125275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:40.7574397Z skip: Need at least 1 CUDA device (1.877s) 2022-08-17T13:12:40.7574732Z 2022-08-17T13:12:40.7575283Z ---------------------------------------------------------------------- 2022-08-17T13:12:40.7575580Z Ran 1 test in 1.878s 2022-08-17T13:12:40.7575708Z 2022-08-17T13:12:40.7575787Z OK (skipped=1) 2022-08-17T13:12:40.7575959Z 2022-08-17T13:12:40.7576073Z Generating XML reports... 2022-08-17T13:12:40.7608580Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131238.xml 2022-08-17T13:12:42.0700603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:42.0701228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:42.0702951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:42.0703577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:42.2240368Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:42.2251066Z 2022-08-17T13:12:42.2251391Z Running tests... 2022-08-17T13:12:42.2252041Z ---------------------------------------------------------------------- 2022-08-17T13:12:42.4906239Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2047 2022-08-17T13:12:42.4934468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2048 2022-08-17T13:12:42.4961333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2049 2022-08-17T13:12:42.4994544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2050 2022-08-17T13:12:43.5600319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:43.5600965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:43.5603927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:43.5604564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:43.5623570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:43.5624581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:43.5625309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:43.5625908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:43.5627821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:43.5628527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:43.5629362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:43.5629971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:43.5632510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:43.5633153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:43.5636821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:43.5637499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:43.7186947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:43.7200360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:43.7210400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:43.7359571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:44.9054857Z ok (2.680s) 2022-08-17T13:12:44.9055137Z 2022-08-17T13:12:44.9055632Z ---------------------------------------------------------------------- 2022-08-17T13:12:44.9055897Z Ran 1 test in 2.680s 2022-08-17T13:12:44.9056020Z 2022-08-17T13:12:44.9056331Z OK 2022-08-17T13:12:44.9056430Z 2022-08-17T13:12:44.9056512Z Generating XML reports... 2022-08-17T13:12:44.9088289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131242.xml 2022-08-17T13:12:46.1808349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:46.1809012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:46.1809852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:46.1810424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:46.3318783Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:46.3329148Z 2022-08-17T13:12:46.3329582Z Running tests... 2022-08-17T13:12:46.3330006Z ---------------------------------------------------------------------- 2022-08-17T13:12:46.5960879Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2138 2022-08-17T13:12:46.5988866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2139 2022-08-17T13:12:46.6017812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2140 2022-08-17T13:12:46.6047468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2141 2022-08-17T13:12:47.6688787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:47.6689409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:47.6690256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:47.6690884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:47.6691663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:47.6692219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:47.6693466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:47.6694133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:47.7345061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:47.7345796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:47.7347144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:47.7347891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:47.8040148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:47.8041266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:47.8042777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:47.8043461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:47.8276422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:47.8305425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:47.8954034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:47.9618045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:49.0109220Z ok (2.678s) 2022-08-17T13:12:49.0109514Z 2022-08-17T13:12:49.0110008Z ---------------------------------------------------------------------- 2022-08-17T13:12:49.0110755Z Ran 1 test in 2.678s 2022-08-17T13:12:49.0110957Z 2022-08-17T13:12:49.0111059Z OK 2022-08-17T13:12:49.0111214Z 2022-08-17T13:12:49.0111342Z Generating XML reports... 2022-08-17T13:12:49.0145063Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131246.xml 2022-08-17T13:12:50.3340764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:50.3341155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:50.3343198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:50.3343671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:50.4856471Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:50.4866253Z 2022-08-17T13:12:50.4866538Z Running tests... 2022-08-17T13:12:50.4867219Z ---------------------------------------------------------------------- 2022-08-17T13:12:50.7476783Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2229 2022-08-17T13:12:50.7505063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2230 2022-08-17T13:12:50.7532417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2231 2022-08-17T13:12:50.7562820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2232 2022-08-17T13:12:51.7773304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:51.7774706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:51.7775626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:51.7776302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:51.8139532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:51.8140182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:51.8142289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:51.8142926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:51.9335031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:51.9576581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:51.9577123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:51.9579518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:51.9580215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:51.9697860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:51.9927325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:51.9927896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:51.9930166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:51.9930718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:52.1349569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:52.2284994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:52.4609148Z skip: Need at least 2 CUDA devices (1.974s) 2022-08-17T13:12:52.4609450Z 2022-08-17T13:12:52.4609931Z ---------------------------------------------------------------------- 2022-08-17T13:12:52.4610198Z Ran 1 test in 1.974s 2022-08-17T13:12:52.4610325Z 2022-08-17T13:12:52.4610403Z OK (skipped=1) 2022-08-17T13:12:52.4610517Z 2022-08-17T13:12:52.4610608Z Generating XML reports... 2022-08-17T13:12:52.4642697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131250.xml 2022-08-17T13:12:53.7522442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:53.7522915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:53.7524629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:53.7525004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:53.9084981Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:53.9095980Z 2022-08-17T13:12:53.9096125Z Running tests... 2022-08-17T13:12:53.9096564Z ---------------------------------------------------------------------- 2022-08-17T13:12:54.1769467Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2284 2022-08-17T13:12:54.1796504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2285 2022-08-17T13:12:54.1825200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2286 2022-08-17T13:12:54.1856035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2287 2022-08-17T13:12:55.2514351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:55.2515127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:55.2515931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:55.2516571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:55.2942730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:55.2943264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:55.2944617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:55.2945150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:55.2955809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:55.2956320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:55.2959748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:55.2960275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:55.3152689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:55.3153338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:55.3154412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:55.3155082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:55.4087702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:55.4528465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:55.4543819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:55.4696553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:55.8905695Z ok (1.981s) 2022-08-17T13:12:55.8905879Z 2022-08-17T13:12:55.8906276Z ---------------------------------------------------------------------- 2022-08-17T13:12:55.8906725Z Ran 1 test in 1.981s 2022-08-17T13:12:55.8906944Z 2022-08-17T13:12:55.8907040Z OK 2022-08-17T13:12:55.8907210Z 2022-08-17T13:12:55.8907340Z Generating XML reports... 2022-08-17T13:12:55.8940765Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131253.xml 2022-08-17T13:12:57.1973999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:57.1974400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:57.1976082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:57.1976489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:57.3487208Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:12:57.3497768Z 2022-08-17T13:12:57.3498078Z Running tests... 2022-08-17T13:12:57.3498770Z ---------------------------------------------------------------------- 2022-08-17T13:12:57.6134666Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2351 2022-08-17T13:12:57.6164506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2352 2022-08-17T13:12:57.6192818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2353 2022-08-17T13:12:57.6225000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2354 2022-08-17T13:12:58.6279828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:58.6280496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:58.6282136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:58.6282770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:58.6433743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:58.6434471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:58.6445466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:58.6446584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:58.6741840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:58.6742513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:58.6744121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:58.6744721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:58.6746665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:12:58.6747273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:12:58.6751133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:12:58.6751780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:12:58.7868967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:12:58.8000087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:12:58.8328621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:12:58.8330564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:12:59.2272414Z ok (1.877s) 2022-08-17T13:12:59.2272685Z 2022-08-17T13:12:59.2273266Z ---------------------------------------------------------------------- 2022-08-17T13:12:59.2273584Z Ran 1 test in 1.877s 2022-08-17T13:12:59.2273705Z 2022-08-17T13:12:59.2273771Z OK 2022-08-17T13:12:59.2273870Z 2022-08-17T13:12:59.2273973Z Generating XML reports... 2022-08-17T13:12:59.2306734Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131257.xml 2022-08-17T13:13:00.5603542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:00.5603954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:00.5605490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:00.5606022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:00.7147914Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:00.7157905Z 2022-08-17T13:13:00.7157993Z Running tests... 2022-08-17T13:13:00.7158967Z ---------------------------------------------------------------------- 2022-08-17T13:13:00.9797845Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2418 2022-08-17T13:13:00.9825624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2419 2022-08-17T13:13:00.9853323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2420 2022-08-17T13:13:00.9884301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2421 2022-08-17T13:13:02.0449170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:02.0449794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:02.0450625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:02.0451197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:02.0451941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:02.0452575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:02.0453367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:02.0453922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:02.0677576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:02.0678227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:02.0679350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:02.0679963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:02.0842052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:02.0842629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:02.0843293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:02.0843818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:02.2047082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:02.2055443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:02.2269061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:02.2438455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:02.5932968Z skip: Need at least 2 CUDA devices (1.877s) 2022-08-17T13:13:02.5933279Z 2022-08-17T13:13:02.5933793Z ---------------------------------------------------------------------- 2022-08-17T13:13:02.5934043Z Ran 1 test in 1.877s 2022-08-17T13:13:02.5934168Z 2022-08-17T13:13:02.5934245Z OK (skipped=1) 2022-08-17T13:13:02.5934359Z 2022-08-17T13:13:02.5934450Z Generating XML reports... 2022-08-17T13:13:02.5970389Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131300.xml 2022-08-17T13:13:03.9142679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:03.9143106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:03.9145271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:03.9145945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:04.0717311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:04.0727407Z 2022-08-17T13:13:04.0727534Z Running tests... 2022-08-17T13:13:04.0728178Z ---------------------------------------------------------------------- 2022-08-17T13:13:04.3397044Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2473 2022-08-17T13:13:04.3425837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2474 2022-08-17T13:13:04.3455506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2475 2022-08-17T13:13:04.3485645Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2476 2022-08-17T13:13:05.4597382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:05.4598069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:05.4599531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:05.4600171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:05.4792315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:05.4792984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:05.4794567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:05.4795170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:05.4815899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:05.4816579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:05.4820099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:05.4820781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:05.5107927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:05.5109354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:05.5110657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:05.5111863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:05.6372751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:05.6396796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:05.6400446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:05.6726467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:06.1536646Z ok (2.080s) 2022-08-17T13:13:06.1536784Z 2022-08-17T13:13:06.1537144Z ---------------------------------------------------------------------- 2022-08-17T13:13:06.1537506Z Ran 1 test in 2.081s 2022-08-17T13:13:06.1537628Z 2022-08-17T13:13:06.1537692Z OK 2022-08-17T13:13:06.1537789Z 2022-08-17T13:13:06.1537872Z Generating XML reports... 2022-08-17T13:13:06.1570320Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131304.xml 2022-08-17T13:13:07.4568209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:07.4568610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:07.4570614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:07.4571039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:07.6131813Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:07.6141478Z 2022-08-17T13:13:07.6141588Z Running tests... 2022-08-17T13:13:07.6142737Z ---------------------------------------------------------------------- 2022-08-17T13:13:07.8770301Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2540 2022-08-17T13:13:07.8799386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2541 2022-08-17T13:13:07.8827574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2542 2022-08-17T13:13:07.8859852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2543 2022-08-17T13:13:08.9092777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:08.9093501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:08.9094740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:08.9095322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:08.9403351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:08.9404052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:08.9405165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:08.9406031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:08.9440988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:08.9441677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:08.9444701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:08.9445297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:08.9680204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:08.9680910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:08.9682373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:08.9682962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:09.0667022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:09.0982712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:09.1043151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:09.1248439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:09.8914484Z ok (2.277s) 2022-08-17T13:13:09.8914693Z 2022-08-17T13:13:09.8915035Z ---------------------------------------------------------------------- 2022-08-17T13:13:09.8915292Z Ran 1 test in 2.277s 2022-08-17T13:13:09.8915415Z 2022-08-17T13:13:09.8915481Z OK 2022-08-17T13:13:09.8915579Z 2022-08-17T13:13:09.8915694Z Generating XML reports... 2022-08-17T13:13:09.8951959Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131307.xml 2022-08-17T13:13:11.2019521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:11.2020209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:11.2021695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:11.2022073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:11.3560154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:11.3571461Z 2022-08-17T13:13:11.3571561Z Running tests... 2022-08-17T13:13:11.3572038Z ---------------------------------------------------------------------- 2022-08-17T13:13:11.6237279Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2631 2022-08-17T13:13:11.6265111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2632 2022-08-17T13:13:11.6293651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2633 2022-08-17T13:13:11.6323648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2634 2022-08-17T13:13:12.6978022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:12.6978651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:12.6980235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:12.6980829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:12.7065215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:12.7065898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:12.7068062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:12.7068714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:12.7113793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:12.7114395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:12.7116516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:12.7117102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:12.7142449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:12.7143522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:12.7146194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:12.7146851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:12.8589747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:12.8695175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:12.8723161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:12.8974129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:13.2374891Z skip: Need at least 2 CUDA devices (1.880s) 2022-08-17T13:13:13.2375286Z 2022-08-17T13:13:13.2375805Z ---------------------------------------------------------------------- 2022-08-17T13:13:13.2376305Z Ran 1 test in 1.880s 2022-08-17T13:13:13.2376508Z 2022-08-17T13:13:13.2376627Z OK (skipped=1) 2022-08-17T13:13:13.2376742Z 2022-08-17T13:13:13.2376820Z Generating XML reports... 2022-08-17T13:13:13.2412608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131311.xml 2022-08-17T13:13:14.5655508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:14.5655919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:14.5656991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:14.5657595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:14.7161718Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:14.7171660Z 2022-08-17T13:13:14.7171810Z Running tests... 2022-08-17T13:13:14.7172403Z ---------------------------------------------------------------------- 2022-08-17T13:13:14.9801450Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2686 2022-08-17T13:13:14.9829731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2687 2022-08-17T13:13:14.9857543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2688 2022-08-17T13:13:14.9886832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2689 2022-08-17T13:13:16.0415914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:16.0416555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:16.0418135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:16.0418789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:16.0970881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:16.0971420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:16.0973693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:16.0974181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:16.1215477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:16.1216126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:16.1217592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:16.1218196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:16.1243499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:16.1244173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:16.1247208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:16.1247858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:16.1999027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:16.2569198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:16.2784850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:16.2819719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:16.6936621Z ok (1.976s) 2022-08-17T13:13:16.6936844Z 2022-08-17T13:13:16.6937395Z ---------------------------------------------------------------------- 2022-08-17T13:13:16.6937763Z Ran 1 test in 1.976s 2022-08-17T13:13:16.6937885Z 2022-08-17T13:13:16.6937949Z OK 2022-08-17T13:13:16.6938045Z 2022-08-17T13:13:16.6938140Z Generating XML reports... 2022-08-17T13:13:16.6974022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131314.xml 2022-08-17T13:13:17.9685683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:17.9686306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:17.9687858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:17.9688262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:18.1233414Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:18.1243109Z 2022-08-17T13:13:18.1243224Z Running tests... 2022-08-17T13:13:18.1243802Z ---------------------------------------------------------------------- 2022-08-17T13:13:18.3898273Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2753 2022-08-17T13:13:18.3925132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2754 2022-08-17T13:13:18.3952704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2755 2022-08-17T13:13:18.3983039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2756 2022-08-17T13:13:19.4974321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:19.4974965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:19.4975918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:19.4976530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:19.5005023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:19.5005719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:19.5008380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:19.5008982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:19.5467389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:19.5467970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:19.5469090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:19.5469926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:19.5792455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:19.5793124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:19.5794079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:19.5794701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:19.6573426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:19.6617251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:19.7055667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:19.7349502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:20.1031755Z ok (1.978s) 2022-08-17T13:13:20.1031999Z 2022-08-17T13:13:20.1032509Z ---------------------------------------------------------------------- 2022-08-17T13:13:20.1032775Z Ran 1 test in 1.979s 2022-08-17T13:13:20.1032897Z 2022-08-17T13:13:20.1033187Z OK 2022-08-17T13:13:20.1033287Z 2022-08-17T13:13:20.1033368Z Generating XML reports... 2022-08-17T13:13:20.1095573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131318.xml 2022-08-17T13:13:21.3909579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:21.3910050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:21.3910730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:21.3911108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:21.5446223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:21.5456586Z 2022-08-17T13:13:21.5456687Z Running tests... 2022-08-17T13:13:21.5457621Z ---------------------------------------------------------------------- 2022-08-17T13:13:21.8085201Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2820 2022-08-17T13:13:21.8111487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2821 2022-08-17T13:13:21.8139139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2822 2022-08-17T13:13:21.8170963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2823 2022-08-17T13:13:22.8311799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:22.8312463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:22.8313525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:22.8314168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:22.8794667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:22.8795364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:22.8797094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:22.8797745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:22.8820793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:22.8821411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:22.8824572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:22.8825332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:22.9159779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:22.9160960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:22.9161861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:22.9162802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:22.9883107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:23.0388389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:23.0424709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:23.0823620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:23.4218850Z skip: Need at least 2 CUDA devices (1.876s) 2022-08-17T13:13:23.4219184Z 2022-08-17T13:13:23.4219969Z ---------------------------------------------------------------------- 2022-08-17T13:13:23.4220237Z Ran 1 test in 1.876s 2022-08-17T13:13:23.4220344Z 2022-08-17T13:13:23.4220423Z OK (skipped=1) 2022-08-17T13:13:23.4220538Z 2022-08-17T13:13:23.4220627Z Generating XML reports... 2022-08-17T13:13:23.4253319Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131321.xml 2022-08-17T13:13:24.7500927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:24.7501321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:24.7503589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:24.7503986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:24.9057603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:24.9067889Z 2022-08-17T13:13:24.9068242Z Running tests... 2022-08-17T13:13:24.9068890Z ---------------------------------------------------------------------- 2022-08-17T13:13:25.1743713Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2875 2022-08-17T13:13:25.1772696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2876 2022-08-17T13:13:25.1799920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2877 2022-08-17T13:13:25.1831918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2878 2022-08-17T13:13:26.2012687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:26.2013351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:26.2015032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:26.2015588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:26.2660937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:26.2661682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:26.2663727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:26.2664448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:26.3190554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:26.3191551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:26.3192380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:26.3193069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:26.3456088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:26.3456883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:26.3457893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:26.3458683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:26.3588927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:26.4311112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:26.4770315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:26.5122004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:26.8879907Z ok (1.981s) 2022-08-17T13:13:26.8880101Z 2022-08-17T13:13:26.8880482Z ---------------------------------------------------------------------- 2022-08-17T13:13:26.8880733Z Ran 1 test in 1.981s 2022-08-17T13:13:26.8880855Z 2022-08-17T13:13:26.8880918Z OK 2022-08-17T13:13:26.8881018Z 2022-08-17T13:13:26.8881180Z Generating XML reports... 2022-08-17T13:13:26.8914362Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131324.xml 2022-08-17T13:13:28.1897404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:28.1897891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:28.1899392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:28.1899773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:28.3440927Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:28.3451105Z 2022-08-17T13:13:28.3451323Z Running tests... 2022-08-17T13:13:28.3451673Z ---------------------------------------------------------------------- 2022-08-17T13:13:28.6164231Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2942 2022-08-17T13:13:28.6193375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2943 2022-08-17T13:13:28.6222245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2944 2022-08-17T13:13:28.6252559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2945 2022-08-17T13:13:29.6617026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:29.6617721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:29.6619057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:29.6619659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:29.6622856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:29.6623543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:29.6626923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:29.6627858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:29.6650697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:29.6651297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:29.6653832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:29.6654479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:29.6879824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:29.6880457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:29.6881734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:29.6882302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:29.8206838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:29.8236677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:29.8237411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:29.8458451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:30.2300692Z ok (1.885s) 2022-08-17T13:13:30.2300932Z 2022-08-17T13:13:30.2301285Z ---------------------------------------------------------------------- 2022-08-17T13:13:30.2301587Z Ran 1 test in 1.885s 2022-08-17T13:13:30.2301711Z 2022-08-17T13:13:30.2301775Z OK 2022-08-17T13:13:30.2301871Z 2022-08-17T13:13:30.2301970Z Generating XML reports... 2022-08-17T13:13:30.2335426Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131328.xml 2022-08-17T13:13:31.5721453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:31.5721995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:31.5723118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:31.5723480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:31.7262275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:31.7272511Z 2022-08-17T13:13:31.7272626Z Running tests... 2022-08-17T13:13:31.7273141Z ---------------------------------------------------------------------- 2022-08-17T13:13:31.9897047Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3009 2022-08-17T13:13:31.9924970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3010 2022-08-17T13:13:31.9952964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3011 2022-08-17T13:13:31.9982307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3012 2022-08-17T13:13:33.0279865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:33.0280511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:33.0281298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:33.0281883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:33.0729939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:33.0730561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:33.0732287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:33.0733264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:33.0905683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:33.0906343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:33.0907552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:33.0908171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:33.1033390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:33.1034070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:33.1035383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:33.1036050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:33.1843991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:33.2316460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:33.2468737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:33.2589968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:35.4062461Z ok (3.679s) 2022-08-17T13:13:35.4062708Z 2022-08-17T13:13:35.4063228Z ---------------------------------------------------------------------- 2022-08-17T13:13:35.4063703Z Ran 1 test in 3.679s 2022-08-17T13:13:35.4063828Z 2022-08-17T13:13:35.4063893Z OK 2022-08-17T13:13:35.4063993Z 2022-08-17T13:13:35.4064089Z Generating XML reports... 2022-08-17T13:13:35.4095837Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131331.xml 2022-08-17T13:13:36.6783154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:36.6783623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:36.6784485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:36.6784900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:36.8302576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:36.8312325Z 2022-08-17T13:13:36.8312406Z Running tests... 2022-08-17T13:13:36.8313128Z ---------------------------------------------------------------------- 2022-08-17T13:13:37.0889442Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3100 2022-08-17T13:13:37.0917063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3101 2022-08-17T13:13:37.0945656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3102 2022-08-17T13:13:37.0975144Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3103 2022-08-17T13:13:38.1585264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:38.1585906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:38.1587135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:38.1587805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:38.1619872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:38.1620803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:38.1623149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:38.1623879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:38.1849756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:38.1850406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:38.1852419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:38.1853055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:38.2310386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:38.2311122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:38.2312267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:38.2312997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:38.3167235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:38.3181137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:38.3419198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:38.3873985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:38.7023544Z skip: Need at least 2 CUDA devices (1.871s) 2022-08-17T13:13:38.7023858Z 2022-08-17T13:13:38.7024384Z ---------------------------------------------------------------------- 2022-08-17T13:13:38.7024645Z Ran 1 test in 1.871s 2022-08-17T13:13:38.7024780Z 2022-08-17T13:13:38.7024856Z OK (skipped=1) 2022-08-17T13:13:38.7024969Z 2022-08-17T13:13:38.7025044Z Generating XML reports... 2022-08-17T13:13:38.7057571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131336.xml 2022-08-17T13:13:39.9902426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:39.9903069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:39.9904764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:39.9905363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:40.1422935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:40.1433481Z 2022-08-17T13:13:40.1433813Z Running tests... 2022-08-17T13:13:40.1434480Z ---------------------------------------------------------------------- 2022-08-17T13:13:40.4026604Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3155 2022-08-17T13:13:40.4054222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3156 2022-08-17T13:13:40.4083013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3157 2022-08-17T13:13:40.4113686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3158 2022-08-17T13:13:41.4899478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:41.4900187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:41.4901032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:41.4901654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:41.5262964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:41.5263747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:41.5264612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:41.5265214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:41.5345490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:41.5346131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:41.5347660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:41.5348222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:41.5469906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:41.5470609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:41.5472264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:41.5473180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:41.6491230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:41.6849548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:41.6940040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:41.7063436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:42.1162767Z ok (1.973s) 2022-08-17T13:13:42.1162974Z 2022-08-17T13:13:42.1163418Z ---------------------------------------------------------------------- 2022-08-17T13:13:42.1163698Z Ran 1 test in 1.973s 2022-08-17T13:13:42.1163881Z 2022-08-17T13:13:42.1163985Z OK 2022-08-17T13:13:42.1164184Z 2022-08-17T13:13:42.1164359Z Generating XML reports... 2022-08-17T13:13:42.1199541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131340.xml 2022-08-17T13:13:43.4083171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:43.4083568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:43.4085480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:43.4086020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:43.5614099Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:43.5624221Z 2022-08-17T13:13:43.5624362Z Running tests... 2022-08-17T13:13:43.5625334Z ---------------------------------------------------------------------- 2022-08-17T13:13:43.8235314Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3226 2022-08-17T13:13:43.8262216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3227 2022-08-17T13:13:43.8289057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3228 2022-08-17T13:13:43.8319365Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3229 2022-08-17T13:13:44.8884828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:44.8885486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:44.8886543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:44.8887438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:44.8890435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:44.8891122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:44.8895041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:44.8895657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:44.8910027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:44.8910700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:44.8914577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:44.8915272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:44.9104064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:44.9104703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:44.9105821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:44.9106437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:45.0479142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:45.0493063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:45.0513082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:45.0668782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:45.5367278Z ok (1.974s) 2022-08-17T13:13:45.5367509Z 2022-08-17T13:13:45.5368036Z ---------------------------------------------------------------------- 2022-08-17T13:13:45.5368494Z Ran 1 test in 1.974s 2022-08-17T13:13:45.5368634Z 2022-08-17T13:13:45.5368697Z OK 2022-08-17T13:13:45.5368793Z 2022-08-17T13:13:45.5368893Z Generating XML reports... 2022-08-17T13:13:45.5405719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131343.xml 2022-08-17T13:13:46.8444648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:46.8445289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:46.8447170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:46.8447763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:46.9984960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:46.9995367Z 2022-08-17T13:13:46.9995596Z Running tests... 2022-08-17T13:13:46.9996263Z ---------------------------------------------------------------------- 2022-08-17T13:13:47.2676033Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3293 2022-08-17T13:13:47.2704178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3294 2022-08-17T13:13:47.2732102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3295 2022-08-17T13:13:47.2761711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3296 2022-08-17T13:13:48.4130301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:48.4130969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:48.4132347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:48.4133003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:48.4174718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:48.4175355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:48.4178715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:48.4179325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:48.4180084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:48.4180684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:48.4181521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:48.4182187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:48.4338481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:48.4339148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:48.4340450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:48.4341101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:48.5713390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:48.5748323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:48.5760070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:48.5885627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:48.8809368Z skip: Need at least 2 CUDA devices (1.881s) 2022-08-17T13:13:48.8809673Z 2022-08-17T13:13:48.8810148Z ---------------------------------------------------------------------- 2022-08-17T13:13:48.8810580Z Ran 1 test in 1.881s 2022-08-17T13:13:48.8810773Z 2022-08-17T13:13:48.8810883Z OK (skipped=1) 2022-08-17T13:13:48.8811055Z 2022-08-17T13:13:48.8811196Z Generating XML reports... 2022-08-17T13:13:48.8845186Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131346.xml 2022-08-17T13:13:50.1945011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:50.1945406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:50.1947103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:50.1947548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:50.3498166Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:50.3507825Z 2022-08-17T13:13:50.3507928Z Running tests... 2022-08-17T13:13:50.3508373Z ---------------------------------------------------------------------- 2022-08-17T13:13:50.6117098Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3348 2022-08-17T13:13:50.6144686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3349 2022-08-17T13:13:50.6173034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3350 2022-08-17T13:13:50.6203380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3351 2022-08-17T13:13:51.7192572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:51.7193522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:51.7194297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:51.7194914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:51.7229677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:51.7230323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:51.7233158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:51.7233782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:51.7689200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:51.7689862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:51.7690997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:51.7691873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:51.7718406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:51.7719092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:51.7721029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:51.7721673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:51.8769392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:51.8805313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:51.9273442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:51.9273965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:52.3251553Z ok (1.974s) 2022-08-17T13:13:52.3251846Z 2022-08-17T13:13:52.3252394Z ---------------------------------------------------------------------- 2022-08-17T13:13:52.3252685Z Ran 1 test in 1.974s 2022-08-17T13:13:52.3252810Z 2022-08-17T13:13:52.3252871Z OK 2022-08-17T13:13:52.3252967Z 2022-08-17T13:13:52.3253062Z Generating XML reports... 2022-08-17T13:13:52.3286030Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131350.xml 2022-08-17T13:13:53.6223217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:53.6223967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:53.6224762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:53.6225351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:53.7758006Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:53.7768213Z 2022-08-17T13:13:53.7768573Z Running tests... 2022-08-17T13:13:53.7769175Z ---------------------------------------------------------------------- 2022-08-17T13:13:54.0397477Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3415 2022-08-17T13:13:54.0424952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3416 2022-08-17T13:13:54.0453397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3417 2022-08-17T13:13:54.0483613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3418 2022-08-17T13:13:55.1230872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:55.1231520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:55.1233597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:55.1234203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:55.1243756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:55.1244565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:55.1248146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:55.1248936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:55.1339478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:55.1340142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:55.1343634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:55.1344276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:55.1832159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:55.1832843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:55.1833907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:55.1834554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:55.2799115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:13:55.2830351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:13:55.2913621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:13:55.3383567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:13:57.2559045Z ok (3.479s) 2022-08-17T13:13:57.2559294Z 2022-08-17T13:13:57.2559850Z ---------------------------------------------------------------------- 2022-08-17T13:13:57.2560258Z Ran 1 test in 3.479s 2022-08-17T13:13:57.2560367Z 2022-08-17T13:13:57.2560434Z OK 2022-08-17T13:13:57.2560528Z 2022-08-17T13:13:57.2560624Z Generating XML reports... 2022-08-17T13:13:57.2595226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131353.xml 2022-08-17T13:13:58.5792180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:58.5792680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:58.5793190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:58.5793658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:13:58.7424556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:13:58.7435136Z 2022-08-17T13:13:58.7435257Z Running tests... 2022-08-17T13:13:58.7435719Z ---------------------------------------------------------------------- 2022-08-17T13:13:59.0092657Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3506 2022-08-17T13:13:59.0121160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3507 2022-08-17T13:13:59.0150932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3508 2022-08-17T13:13:59.0182244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3509 2022-08-17T13:13:59.9991154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:13:59.9991820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:13:59.9993455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:13:59.9994085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:00.0163137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:00.0163789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:00.0165140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:00.0165758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:00.0632161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:00.0633280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:00.0634331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:00.0634912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:00.0773778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:00.0774496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:00.0776134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:00.0776786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:00.1568250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:00.1732122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:00.2203323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:00.2424725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:00.5228345Z skip: Need at least 2 CUDA devices (1.779s) 2022-08-17T13:14:00.5228625Z 2022-08-17T13:14:00.5229151Z ---------------------------------------------------------------------- 2022-08-17T13:14:00.5229600Z Ran 1 test in 1.779s 2022-08-17T13:14:00.5229742Z 2022-08-17T13:14:00.5229810Z OK (skipped=1) 2022-08-17T13:14:00.5229922Z 2022-08-17T13:14:00.5230013Z Generating XML reports... 2022-08-17T13:14:00.5263216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131358.xml 2022-08-17T13:14:01.8295681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:01.8296162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:01.8297790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:01.8298154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:01.9847652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:01.9858619Z 2022-08-17T13:14:01.9858993Z Running tests... 2022-08-17T13:14:01.9859400Z ---------------------------------------------------------------------- 2022-08-17T13:14:02.2491066Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3561 2022-08-17T13:14:02.2518643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3562 2022-08-17T13:14:02.2546159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3563 2022-08-17T13:14:02.2575173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3564 2022-08-17T13:14:03.3518607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:03.3519462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:03.3520857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:03.3521552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:03.3838366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:03.3838996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:03.3840521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:03.3841093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:03.3948373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:03.3949041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:03.3950377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:03.3951024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:03.3952175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:03.3952803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:03.3956475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:03.3957099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:03.5084001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:03.5429454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:03.5511624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:03.5533616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:04.0626183Z ok (2.076s) 2022-08-17T13:14:04.0626419Z 2022-08-17T13:14:04.0626952Z ---------------------------------------------------------------------- 2022-08-17T13:14:04.0627327Z Ran 1 test in 2.077s 2022-08-17T13:14:04.0627448Z 2022-08-17T13:14:04.0627510Z OK 2022-08-17T13:14:04.0627604Z 2022-08-17T13:14:04.0627697Z Generating XML reports... 2022-08-17T13:14:04.0661086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131401.xml 2022-08-17T13:14:05.3917501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:05.3917902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:05.3919711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:05.3920087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:05.5440871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:05.5451684Z 2022-08-17T13:14:05.5451997Z Running tests... 2022-08-17T13:14:05.5452669Z ---------------------------------------------------------------------- 2022-08-17T13:14:05.8081110Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3640 2022-08-17T13:14:05.8109530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3641 2022-08-17T13:14:05.8136749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3642 2022-08-17T13:14:05.8167301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3643 2022-08-17T13:14:06.9351942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:06.9352753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:06.9353554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:06.9354163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:06.9516109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:06.9516771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:06.9519541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:06.9520451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:06.9867787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:06.9868907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:06.9870173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:06.9870859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:07.0063476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:07.0064095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:07.0065583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:07.0066106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:07.1096335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:07.1117154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:07.1484530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:07.1648268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:07.8222077Z ok (2.277s) 2022-08-17T13:14:07.8222338Z 2022-08-17T13:14:07.8222829Z ---------------------------------------------------------------------- 2022-08-17T13:14:07.8223327Z Ran 1 test in 2.277s 2022-08-17T13:14:07.8223523Z 2022-08-17T13:14:07.8223622Z OK 2022-08-17T13:14:07.8223766Z 2022-08-17T13:14:07.8223911Z Generating XML reports... 2022-08-17T13:14:07.8257008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131405.xml 2022-08-17T13:14:09.1421513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:09.1421947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:09.1423372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:09.1423747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:09.2952260Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:09.2963119Z 2022-08-17T13:14:09.2963383Z Running tests... 2022-08-17T13:14:09.2964336Z ---------------------------------------------------------------------- 2022-08-17T13:14:09.5594418Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3743 2022-08-17T13:14:09.5622167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3744 2022-08-17T13:14:09.5649510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3745 2022-08-17T13:14:09.5679664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3746 2022-08-17T13:14:10.6285168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:10.6286056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:10.6286859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:10.6287434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:10.6290684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:10.6291327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:10.6292412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:10.6293021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:10.6379010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:10.6379635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:10.6382211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:10.6382798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:10.6563758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:10.6564450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:10.6567152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:10.6567836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:10.7876191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:10.7876906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:10.7974954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:10.8135562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:11.1727320Z ok (1.876s) 2022-08-17T13:14:11.1729478Z 2022-08-17T13:14:11.1729948Z ---------------------------------------------------------------------- 2022-08-17T13:14:11.1730238Z Ran 1 test in 1.876s 2022-08-17T13:14:11.1730358Z 2022-08-17T13:14:11.1730421Z OK 2022-08-17T13:14:11.1730507Z 2022-08-17T13:14:11.1737235Z Generating XML reports... 2022-08-17T13:14:11.1761839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131409.xml 2022-08-17T13:14:12.4814861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:12.4815712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:12.4841515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:12.4841915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:12.6363657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:12.6373106Z 2022-08-17T13:14:12.6373189Z Running tests... 2022-08-17T13:14:12.6374060Z ---------------------------------------------------------------------- 2022-08-17T13:14:12.9016430Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3810 2022-08-17T13:14:12.9043924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3811 2022-08-17T13:14:12.9072198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3812 2022-08-17T13:14:12.9109057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3813 2022-08-17T13:14:13.9269235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:13.9269865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:13.9271191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:13.9271843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:13.9462044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:13.9462980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:13.9463865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:13.9464431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:13.9465225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:13.9465853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:13.9466617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:13.9467237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:13.9702507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:13.9703252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:13.9704703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:13.9705249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:14.0843316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:14.1030755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:14.1048770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:14.1281923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:14.4155714Z skip: Need at least 2 CUDA devices (1.778s) 2022-08-17T13:14:14.4156040Z 2022-08-17T13:14:14.4156567Z ---------------------------------------------------------------------- 2022-08-17T13:14:14.4156842Z Ran 1 test in 1.778s 2022-08-17T13:14:14.4156978Z 2022-08-17T13:14:14.4157054Z OK (skipped=1) 2022-08-17T13:14:14.4157171Z 2022-08-17T13:14:14.4157262Z Generating XML reports... 2022-08-17T13:14:14.4192577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131412.xml 2022-08-17T13:14:15.7294968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:15.7295425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:15.7296985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:15.7297648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:15.8828611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:15.8838651Z 2022-08-17T13:14:15.8838730Z Running tests... 2022-08-17T13:14:15.8839306Z ---------------------------------------------------------------------- 2022-08-17T13:14:16.1487011Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3865 2022-08-17T13:14:16.1514620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3866 2022-08-17T13:14:16.1543253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3867 2022-08-17T13:14:16.1574416Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3868 2022-08-17T13:14:17.2243270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:17.2243924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:17.2245128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:17.2245693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:17.2379183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:17.2379856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:17.2381107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:17.2381811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:17.2390050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:17.2390711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:17.2394969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:17.2395620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:17.2741966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:17.2742509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:17.2743475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:17.2744015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:17.3864311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:17.3981098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:17.4071317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:17.4329681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:17.8624411Z ok (1.978s) 2022-08-17T13:14:17.8624594Z 2022-08-17T13:14:17.8625001Z ---------------------------------------------------------------------- 2022-08-17T13:14:17.8625253Z Ran 1 test in 1.978s 2022-08-17T13:14:17.8625379Z 2022-08-17T13:14:17.8625447Z OK 2022-08-17T13:14:17.8625543Z 2022-08-17T13:14:17.8625636Z Generating XML reports... 2022-08-17T13:14:17.8659137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131415.xml 2022-08-17T13:14:19.1874330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:19.1874723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:19.1876294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:19.1876668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:19.3421644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:19.3431599Z 2022-08-17T13:14:19.3431742Z Running tests... 2022-08-17T13:14:19.3432232Z ---------------------------------------------------------------------- 2022-08-17T13:14:19.6157513Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3932 2022-08-17T13:14:19.6184510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3933 2022-08-17T13:14:19.6211688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3934 2022-08-17T13:14:19.6241442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3935 2022-08-17T13:14:20.6769388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:20.6770079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:20.6771603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:20.6772270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:20.7314511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:20.7315358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:20.7316148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:20.7316749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:20.7524462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:20.7525745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:20.7526989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:20.7527625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:20.7622950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:20.7623708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:20.7625933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:20.7626541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:20.8349154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:20.8880923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:20.9105630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:20.9185113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:23.1322515Z ok (3.789s) 2022-08-17T13:14:23.1322793Z 2022-08-17T13:14:23.1323273Z ---------------------------------------------------------------------- 2022-08-17T13:14:23.1323546Z Ran 1 test in 3.789s 2022-08-17T13:14:23.1323668Z 2022-08-17T13:14:23.1323734Z OK 2022-08-17T13:14:23.1323831Z 2022-08-17T13:14:23.1323926Z Generating XML reports... 2022-08-17T13:14:23.1356570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131419.xml 2022-08-17T13:14:24.4373572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:24.4374319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:24.4375631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:24.4376168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:24.5937267Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:24.5947551Z 2022-08-17T13:14:24.5947691Z Running tests... 2022-08-17T13:14:24.5948186Z ---------------------------------------------------------------------- 2022-08-17T13:14:24.5955082Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s) 2022-08-17T13:14:24.5955478Z 2022-08-17T13:14:24.5955768Z ---------------------------------------------------------------------- 2022-08-17T13:14:24.5956027Z Ran 1 test in 0.001s 2022-08-17T13:14:24.5956159Z 2022-08-17T13:14:24.5956238Z OK (skipped=1) 2022-08-17T13:14:24.5956351Z 2022-08-17T13:14:24.5956431Z Generating XML reports... 2022-08-17T13:14:24.5980746Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131424.xml 2022-08-17T13:14:25.7822158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:25.7822693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:25.7823934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:25.7824474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:25.9383670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:25.9393384Z 2022-08-17T13:14:25.9393691Z Running tests... 2022-08-17T13:14:25.9394328Z ---------------------------------------------------------------------- 2022-08-17T13:14:26.2056686Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4033 2022-08-17T13:14:26.2084161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4034 2022-08-17T13:14:26.2112779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4035 2022-08-17T13:14:26.2142840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4036 2022-08-17T13:14:27.2979101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:27.2979855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:27.2980708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:27.2981335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:27.3199274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:27.3199916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:27.3202036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:27.3202678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:27.3401578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:27.3402220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:27.3403380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:27.3404040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:27.3685509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:27.3686716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:27.3687570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:27.3688234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:27.4557879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:27.4784497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:27.4964555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:27.5236888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:27.9192328Z ok (1.980s) 2022-08-17T13:14:27.9192774Z 2022-08-17T13:14:27.9193326Z ---------------------------------------------------------------------- 2022-08-17T13:14:27.9193678Z Ran 1 test in 1.980s 2022-08-17T13:14:27.9193800Z 2022-08-17T13:14:27.9193861Z OK 2022-08-17T13:14:27.9193945Z 2022-08-17T13:14:27.9194043Z Generating XML reports... 2022-08-17T13:14:27.9230390Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131425.xml 2022-08-17T13:14:29.1956919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:29.1957348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:29.1958850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:29.1959228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:29.3462709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:29.3473345Z 2022-08-17T13:14:29.3473640Z Running tests... 2022-08-17T13:14:29.3474263Z ---------------------------------------------------------------------- 2022-08-17T13:14:29.3477270Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s) 2022-08-17T13:14:29.3477676Z 2022-08-17T13:14:29.3478103Z ---------------------------------------------------------------------- 2022-08-17T13:14:29.3478523Z Ran 1 test in 0.000s 2022-08-17T13:14:29.3478646Z 2022-08-17T13:14:29.3478724Z OK (skipped=1) 2022-08-17T13:14:29.3478826Z 2022-08-17T13:14:29.3478919Z Generating XML reports... 2022-08-17T13:14:29.3502621Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131429.xml 2022-08-17T13:14:30.5063996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:30.5064389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:30.5066381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:30.5066742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:30.6581434Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:30.6591750Z 2022-08-17T13:14:30.6591863Z Running tests... 2022-08-17T13:14:30.6592873Z ---------------------------------------------------------------------- 2022-08-17T13:14:30.9178479Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4110 2022-08-17T13:14:30.9205346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4111 2022-08-17T13:14:30.9236161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4112 2022-08-17T13:14:30.9266932Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4113 2022-08-17T13:14:31.9930287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:31.9930963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:31.9931961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:31.9932566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:32.0127905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:32.0128507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:32.0129437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:32.0129842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:32.0769399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:32.0770062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:32.0771616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:32.0772242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:32.0950440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:32.0951113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:32.0952517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:32.0953162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:32.1567467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:32.1724732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:32.2335301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:32.2515237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:32.5315099Z skip: Need at least 2 CUDA devices (1.872s) 2022-08-17T13:14:32.5315369Z 2022-08-17T13:14:32.5315894Z ---------------------------------------------------------------------- 2022-08-17T13:14:32.5316171Z Ran 1 test in 1.872s 2022-08-17T13:14:32.5316292Z 2022-08-17T13:14:32.5316368Z OK (skipped=1) 2022-08-17T13:14:32.5316480Z 2022-08-17T13:14:32.5316555Z Generating XML reports... 2022-08-17T13:14:32.5349315Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131430.xml 2022-08-17T13:14:33.8052569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:33.8052984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:33.8055578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:33.8055990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:33.9564301Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:33.9574856Z 2022-08-17T13:14:33.9575180Z Running tests... 2022-08-17T13:14:33.9575836Z ---------------------------------------------------------------------- 2022-08-17T13:14:34.2211177Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4165 2022-08-17T13:14:34.2237855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4166 2022-08-17T13:14:34.2265039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4167 2022-08-17T13:14:34.2294138Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4168 2022-08-17T13:14:35.2669716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:35.2670386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:35.2672043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:35.2672611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:35.2866329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:35.2866981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:35.2868676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:35.2869327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:35.3137916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:35.3138822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:35.3140047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:35.3140582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:35.3317261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:35.3318011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:35.3319495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:35.3320132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:35.4233566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:14:35.4436054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:14:35.4712352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:14:35.4892202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:14:35.9344070Z ok (1.977s) 2022-08-17T13:14:35.9344292Z 2022-08-17T13:14:35.9344735Z ---------------------------------------------------------------------- 2022-08-17T13:14:35.9345144Z Ran 1 test in 1.977s 2022-08-17T13:14:35.9345336Z 2022-08-17T13:14:35.9345431Z OK 2022-08-17T13:14:35.9345592Z 2022-08-17T13:14:35.9345747Z Generating XML reports... 2022-08-17T13:14:35.9379351Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131433.xml 2022-08-17T13:14:37.2246835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:37.2247284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:37.2249129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:37.2249666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:37.3785247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:37.3795659Z 2022-08-17T13:14:37.3795778Z Running tests... 2022-08-17T13:14:37.3796397Z ---------------------------------------------------------------------- 2022-08-17T13:14:37.3840382Z test_forward_backward (__main__.ReducerTest) ... ok (0.004s) 2022-08-17T13:14:37.3912431Z 2022-08-17T13:14:37.3913037Z ---------------------------------------------------------------------- 2022-08-17T13:14:37.3913700Z Ran 1 test in 0.012s 2022-08-17T13:14:37.3913823Z 2022-08-17T13:14:37.3913889Z OK 2022-08-17T13:14:37.3913987Z 2022-08-17T13:14:37.3914083Z Generating XML reports... 2022-08-17T13:14:37.3945788Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131437.xml 2022-08-17T13:14:38.5643968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:38.5644489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:38.5645711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:38.5646352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:38.7156960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:38.7167304Z 2022-08-17T13:14:38.7167405Z Running tests... 2022-08-17T13:14:38.7167934Z ---------------------------------------------------------------------- 2022-08-17T13:14:38.7220526Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-08-17T13:14:38.7233031Z ok (0.006s) 2022-08-17T13:14:38.7285794Z 2022-08-17T13:14:38.7286630Z ---------------------------------------------------------------------- 2022-08-17T13:14:38.7287103Z Ran 1 test in 0.012s 2022-08-17T13:14:38.7287305Z 2022-08-17T13:14:38.7287375Z OK 2022-08-17T13:14:38.7287473Z 2022-08-17T13:14:38.7287567Z Generating XML reports... 2022-08-17T13:14:38.7317814Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131438.xml 2022-08-17T13:14:39.8938499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:39.8939096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:39.8941055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:39.8941666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:40.0452635Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:40.0463382Z 2022-08-17T13:14:40.0463686Z Running tests... 2022-08-17T13:14:40.0464356Z ---------------------------------------------------------------------- 2022-08-17T13:14:40.0510467Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.005s) 2022-08-17T13:14:40.0581519Z 2022-08-17T13:14:40.0582233Z ---------------------------------------------------------------------- 2022-08-17T13:14:40.0582688Z Ran 1 test in 0.012s 2022-08-17T13:14:40.0582866Z 2022-08-17T13:14:40.0582980Z OK 2022-08-17T13:14:40.0583217Z 2022-08-17T13:14:40.0583377Z Generating XML reports... 2022-08-17T13:14:40.0613790Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131440.xml 2022-08-17T13:14:41.2042499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:41.2043131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:41.2045248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:41.2046427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:41.3574289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:41.3584890Z 2022-08-17T13:14:41.3585264Z Running tests... 2022-08-17T13:14:41.3585946Z ---------------------------------------------------------------------- 2022-08-17T13:14:41.3612522Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.003s) 2022-08-17T13:14:41.3700882Z 2022-08-17T13:14:41.3701370Z ---------------------------------------------------------------------- 2022-08-17T13:14:41.3701802Z Ran 1 test in 0.012s 2022-08-17T13:14:41.3701995Z 2022-08-17T13:14:41.3702099Z OK 2022-08-17T13:14:41.3702262Z 2022-08-17T13:14:41.3702423Z Generating XML reports... 2022-08-17T13:14:41.3733051Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131441.xml 2022-08-17T13:14:42.5265749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:42.5266360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:42.5268898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:42.5269814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:42.6778296Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:42.6788663Z 2022-08-17T13:14:42.6789120Z Running tests... 2022-08-17T13:14:42.6789535Z ---------------------------------------------------------------------- 2022-08-17T13:14:42.6839786Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.005s) 2022-08-17T13:14:42.6903714Z 2022-08-17T13:14:42.6904135Z ---------------------------------------------------------------------- 2022-08-17T13:14:42.6904418Z Ran 1 test in 0.011s 2022-08-17T13:14:42.6904541Z 2022-08-17T13:14:42.6904605Z OK 2022-08-17T13:14:42.6904707Z 2022-08-17T13:14:42.6904799Z Generating XML reports... 2022-08-17T13:14:42.6937463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131442.xml 2022-08-17T13:14:43.8478981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:43.8479478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:43.8481056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:43.8481426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:44.0018624Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:44.0028490Z 2022-08-17T13:14:44.0028713Z Running tests... 2022-08-17T13:14:44.0029054Z ---------------------------------------------------------------------- 2022-08-17T13:14:44.0052754Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.002s) 2022-08-17T13:14:44.0142151Z 2022-08-17T13:14:44.0142528Z ---------------------------------------------------------------------- 2022-08-17T13:14:44.0143064Z Ran 1 test in 0.011s 2022-08-17T13:14:44.0143266Z 2022-08-17T13:14:44.0143365Z OK 2022-08-17T13:14:44.0143492Z 2022-08-17T13:14:44.0143588Z Generating XML reports... 2022-08-17T13:14:44.0175495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131443.xml 2022-08-17T13:14:45.1836779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:45.1837311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:45.1838366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:45.1838756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:45.3376056Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:45.3386604Z 2022-08-17T13:14:45.3386736Z Running tests... 2022-08-17T13:14:45.3387174Z ---------------------------------------------------------------------- 2022-08-17T13:14:45.5962185Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:14:45.5962894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T13:14:45.6061444Z ok (0.267s) 2022-08-17T13:14:45.6061908Z 2022-08-17T13:14:45.6062439Z ---------------------------------------------------------------------- 2022-08-17T13:14:45.6062871Z Ran 1 test in 0.267s 2022-08-17T13:14:45.6063139Z 2022-08-17T13:14:45.6063260Z OK 2022-08-17T13:14:45.6063426Z 2022-08-17T13:14:45.6063586Z Generating XML reports... 2022-08-17T13:14:45.6090678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220817131445.xml 2022-08-17T13:14:46.8636803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:46.8637545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:46.8638761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:46.8639225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:47.0191458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-08-17T13:14:47.0201639Z 2022-08-17T13:14:47.0201762Z Running tests... 2022-08-17T13:14:47.0202189Z ---------------------------------------------------------------------- 2022-08-17T13:14:47.2743489Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s) 2022-08-17T13:14:47.2743993Z 2022-08-17T13:14:47.2744215Z ---------------------------------------------------------------------- 2022-08-17T13:14:47.2744491Z Ran 1 test in 0.254s 2022-08-17T13:14:47.2744598Z 2022-08-17T13:14:47.2744675Z OK (skipped=1) 2022-08-17T13:14:47.2744786Z 2022-08-17T13:14:47.2744873Z Generating XML reports... 2022-08-17T13:14:47.2768227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220817131447.xml 2022-08-17T13:14:47.8223415Z Running distributed/test_c10d_nccl ... [2022-08-17 13:14:47.821882] 2022-08-17T13:14:47.8224065Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:14:47.821972] 2022-08-17T13:14:48.9636560Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>]> 2022-08-17T13:14:48.9638276Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-08-17T13:14:48.9638578Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-08-17T13:14:48.9638940Z test_nccl_barrier (__main__.CommTest) 2022-08-17T13:14:48.9639302Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-08-17T13:14:48.9639616Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-08-17T13:14:48.9639890Z test_nccl_barrier_timeout (__main__.CommTest) 2022-08-17T13:14:48.9640147Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-08-17T13:14:48.9640415Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-08-17T13:14:48.9640704Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-08-17T13:14:48.9640980Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-08-17T13:14:48.9641237Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-08-17T13:14:48.9641516Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-08-17T13:14:48.9641802Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-08-17T13:14:48.9642185Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-08-17T13:14:48.9642457Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-08-17T13:14:48.9642725Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-08-17T13:14:48.9650601Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-08-17T13:14:48.9658108Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9658469Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9658838Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9659203Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9659546Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9659887Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9660258Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9660604Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9660933Z test_channels_last_contig (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9661274Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9663454Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9664131Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9664927Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9665420Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9665828Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9666227Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9666588Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9667028Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9667417Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9667813Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9668195Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9668590Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9668967Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9669317Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9669695Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9670064Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9670429Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9670765Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9671093Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9671416Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9671739Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9672081Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9672413Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9672754Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9673108Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9673471Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9673850Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9674233Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9674622Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9674950Z test_fp16 (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9675260Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9675582Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9675903Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9676303Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9676651Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9676985Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9677326Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9677691Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9678059Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9678444Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9678807Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9679130Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9679484Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9679857Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9680268Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9680639Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9681002Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9681314Z test_no_grad (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9681612Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9681936Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9682265Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9682616Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9682952Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9683288Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-08-17T13:14:48.9683581Z 2022-08-17T13:14:48.9684604Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-08-17T13:14:48.9685603Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9686109Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9686420Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9686735Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9687051Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9687349Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9687658Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9687965Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9688235Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-08-17T13:14:48.9688595Z ]> 2022-08-17T13:14:48.9689027Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-08-17T13:14:48.9690603Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]> 2022-08-17T13:14:48.9692278Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9692580Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9692865Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9693127Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9693396Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9693666Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9693939Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9694198Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9694469Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9694740Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9694995Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9695285Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9695589Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9695865Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9696147Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9696415Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9696672Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9696939Z test_send_recv (__main__.ProcessGroupNCCLTest) 2022-08-17T13:14:48.9697271Z ]> 2022-08-17T13:14:48.9697591Z test_common_errors (__main__.RendezvousEnvTest) 2022-08-17T13:14:48.9697827Z 2022-08-17T13:14:48.9698157Z ]> 2022-08-17T13:14:48.9698486Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-08-17T13:14:49.9332758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:49.9333146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:49.9335573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:49.9336005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:50.0873710Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:50.0883939Z 2022-08-17T13:14:50.0884289Z Running tests... 2022-08-17T13:14:50.0884656Z ---------------------------------------------------------------------- 2022-08-17T13:14:50.0895308Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:50.0895756Z 2022-08-17T13:14:50.0896142Z ---------------------------------------------------------------------- 2022-08-17T13:14:50.0896405Z Ran 1 test in 0.001s 2022-08-17T13:14:50.0896525Z 2022-08-17T13:14:50.0896616Z OK (skipped=1) 2022-08-17T13:14:50.0896733Z 2022-08-17T13:14:50.0896811Z Generating XML reports... 2022-08-17T13:14:50.0926581Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131450.xml 2022-08-17T13:14:51.2628071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:51.2628749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:51.2631551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:51.2632158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:51.4183962Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:51.4194105Z 2022-08-17T13:14:51.4194600Z Running tests... 2022-08-17T13:14:51.4195028Z ---------------------------------------------------------------------- 2022-08-17T13:14:51.4202330Z test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:51.4202657Z 2022-08-17T13:14:51.4202904Z ---------------------------------------------------------------------- 2022-08-17T13:14:51.4203251Z Ran 1 test in 0.001s 2022-08-17T13:14:51.4203372Z 2022-08-17T13:14:51.4203449Z OK (skipped=1) 2022-08-17T13:14:51.4203550Z 2022-08-17T13:14:51.4203642Z Generating XML reports... 2022-08-17T13:14:51.4233120Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131451.xml 2022-08-17T13:14:52.5981863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:52.5982264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:52.5984413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:52.5984798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:52.7505307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:52.7515351Z 2022-08-17T13:14:52.7515429Z Running tests... 2022-08-17T13:14:52.7516393Z ---------------------------------------------------------------------- 2022-08-17T13:14:52.7535652Z test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:14:52.7536171Z 2022-08-17T13:14:52.7536522Z ---------------------------------------------------------------------- 2022-08-17T13:14:52.7536768Z Ran 1 test in 0.002s 2022-08-17T13:14:52.7536899Z 2022-08-17T13:14:52.7536978Z OK (skipped=1) 2022-08-17T13:14:52.7537093Z 2022-08-17T13:14:52.7537185Z Generating XML reports... 2022-08-17T13:14:52.7566228Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131452.xml 2022-08-17T13:14:53.9179768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:53.9180456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:53.9183084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:53.9183504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:54.0706722Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:54.0716312Z 2022-08-17T13:14:54.0716436Z Running tests... 2022-08-17T13:14:54.0717200Z ---------------------------------------------------------------------- 2022-08-17T13:14:54.0723082Z test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:14:54.0723526Z 2022-08-17T13:14:54.0723938Z ---------------------------------------------------------------------- 2022-08-17T13:14:54.0724409Z Ran 1 test in 0.001s 2022-08-17T13:14:54.0724563Z 2022-08-17T13:14:54.0724638Z OK (skipped=1) 2022-08-17T13:14:54.0724749Z 2022-08-17T13:14:54.0724838Z Generating XML reports... 2022-08-17T13:14:54.0754382Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131454.xml 2022-08-17T13:14:55.2354215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:55.2354687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:55.2356554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:55.2357087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:55.3869133Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:55.3878839Z 2022-08-17T13:14:55.3878968Z Running tests... 2022-08-17T13:14:55.3879863Z ---------------------------------------------------------------------- 2022-08-17T13:14:55.3887665Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:55.3888128Z 2022-08-17T13:14:55.3888512Z ---------------------------------------------------------------------- 2022-08-17T13:14:55.3888775Z Ran 1 test in 0.001s 2022-08-17T13:14:55.3888894Z 2022-08-17T13:14:55.3888971Z OK (skipped=1) 2022-08-17T13:14:55.3889085Z 2022-08-17T13:14:55.3889174Z Generating XML reports... 2022-08-17T13:14:55.3919310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131455.xml 2022-08-17T13:14:56.5412723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:56.5413096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:56.5415553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:56.5416032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:56.6970360Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:56.6980548Z 2022-08-17T13:14:56.6980671Z Running tests... 2022-08-17T13:14:56.6981155Z ---------------------------------------------------------------------- 2022-08-17T13:14:56.6988801Z test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:56.6989111Z 2022-08-17T13:14:56.6989536Z ---------------------------------------------------------------------- 2022-08-17T13:14:56.6989970Z Ran 1 test in 0.001s 2022-08-17T13:14:56.6990184Z 2022-08-17T13:14:56.6990319Z OK (skipped=1) 2022-08-17T13:14:56.6990519Z 2022-08-17T13:14:56.6990680Z Generating XML reports... 2022-08-17T13:14:56.7020371Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131456.xml 2022-08-17T13:14:57.8818052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:57.8818508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:57.8820141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:57.8820560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:58.0387685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:58.0396949Z 2022-08-17T13:14:58.0397056Z Running tests... 2022-08-17T13:14:58.0397677Z ---------------------------------------------------------------------- 2022-08-17T13:14:58.0408516Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:58.0408768Z 2022-08-17T13:14:58.0409105Z ---------------------------------------------------------------------- 2022-08-17T13:14:58.0409512Z Ran 1 test in 0.001s 2022-08-17T13:14:58.0409699Z 2022-08-17T13:14:58.0409774Z OK (skipped=1) 2022-08-17T13:14:58.0409894Z 2022-08-17T13:14:58.0409987Z Generating XML reports... 2022-08-17T13:14:58.0440935Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131458.xml 2022-08-17T13:14:59.2304944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:14:59.2305363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:14:59.2307636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:14:59.2308156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:14:59.3874165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:14:59.3883863Z 2022-08-17T13:14:59.3884020Z Running tests... 2022-08-17T13:14:59.3884672Z ---------------------------------------------------------------------- 2022-08-17T13:14:59.3894694Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:14:59.3895021Z 2022-08-17T13:14:59.3895309Z ---------------------------------------------------------------------- 2022-08-17T13:14:59.3895612Z Ran 1 test in 0.001s 2022-08-17T13:14:59.3895771Z 2022-08-17T13:14:59.3895852Z OK (skipped=1) 2022-08-17T13:14:59.3895966Z 2022-08-17T13:14:59.3896067Z Generating XML reports... 2022-08-17T13:14:59.3926819Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131459.xml 2022-08-17T13:15:00.5762069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:00.5762492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:00.5764775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:00.5765337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:00.7318032Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:00.7327898Z 2022-08-17T13:15:00.7328040Z Running tests... 2022-08-17T13:15:00.7328438Z ---------------------------------------------------------------------- 2022-08-17T13:15:00.7334355Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:00.7334810Z 2022-08-17T13:15:00.7335266Z ---------------------------------------------------------------------- 2022-08-17T13:15:00.7335651Z Ran 1 test in 0.001s 2022-08-17T13:15:00.7335812Z 2022-08-17T13:15:00.7335926Z OK (skipped=1) 2022-08-17T13:15:00.7336105Z 2022-08-17T13:15:00.7336234Z Generating XML reports... 2022-08-17T13:15:00.7367666Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131500.xml 2022-08-17T13:15:01.9208645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:01.9209014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:01.9211572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:01.9211950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:02.0764445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:02.0774477Z 2022-08-17T13:15:02.0774570Z Running tests... 2022-08-17T13:15:02.0775301Z ---------------------------------------------------------------------- 2022-08-17T13:15:02.0780459Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:02.0780854Z 2022-08-17T13:15:02.0781309Z ---------------------------------------------------------------------- 2022-08-17T13:15:02.0781634Z Ran 1 test in 0.001s 2022-08-17T13:15:02.0781754Z 2022-08-17T13:15:02.0781830Z OK (skipped=1) 2022-08-17T13:15:02.0781943Z 2022-08-17T13:15:02.0782020Z Generating XML reports... 2022-08-17T13:15:02.0814849Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131502.xml 2022-08-17T13:15:03.2749881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:03.2750281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:03.2752446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:03.2753279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:03.4318114Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:03.4328038Z 2022-08-17T13:15:03.4328179Z Running tests... 2022-08-17T13:15:03.4328775Z ---------------------------------------------------------------------- 2022-08-17T13:15:03.4334074Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:03.4334544Z 2022-08-17T13:15:03.4335120Z ---------------------------------------------------------------------- 2022-08-17T13:15:03.4335559Z Ran 1 test in 0.001s 2022-08-17T13:15:03.4335686Z 2022-08-17T13:15:03.4335762Z OK (skipped=1) 2022-08-17T13:15:03.4335876Z 2022-08-17T13:15:03.4335967Z Generating XML reports... 2022-08-17T13:15:03.4366754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131503.xml 2022-08-17T13:15:04.6234208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:04.6234605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:04.6237028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:04.6237408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:04.7813876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:04.7824299Z 2022-08-17T13:15:04.7824596Z Running tests... 2022-08-17T13:15:04.7824965Z ---------------------------------------------------------------------- 2022-08-17T13:15:04.7835355Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:15:04.7835851Z 2022-08-17T13:15:04.7836337Z ---------------------------------------------------------------------- 2022-08-17T13:15:04.7836659Z Ran 1 test in 0.001s 2022-08-17T13:15:04.7836780Z 2022-08-17T13:15:04.7836854Z OK (skipped=1) 2022-08-17T13:15:04.7836969Z 2022-08-17T13:15:04.7837044Z Generating XML reports... 2022-08-17T13:15:04.7867587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131504.xml 2022-08-17T13:15:05.9665086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:05.9665542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:05.9668051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:05.9668708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:06.1200568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:06.1210929Z 2022-08-17T13:15:06.1211371Z Running tests... 2022-08-17T13:15:06.1212020Z ---------------------------------------------------------------------- 2022-08-17T13:15:06.1216964Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:06.1217390Z 2022-08-17T13:15:06.1217853Z ---------------------------------------------------------------------- 2022-08-17T13:15:06.1218134Z Ran 1 test in 0.001s 2022-08-17T13:15:06.1218252Z 2022-08-17T13:15:06.1218329Z OK (skipped=1) 2022-08-17T13:15:06.1218444Z 2022-08-17T13:15:06.1218533Z Generating XML reports... 2022-08-17T13:15:06.1247372Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131506.xml 2022-08-17T13:15:07.2935054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:07.2935453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:07.2937937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:07.2938323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:07.4484126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:07.4494361Z 2022-08-17T13:15:07.4494738Z Running tests... 2022-08-17T13:15:07.4495156Z ---------------------------------------------------------------------- 2022-08-17T13:15:07.4500120Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:07.4500766Z 2022-08-17T13:15:07.4501078Z ---------------------------------------------------------------------- 2022-08-17T13:15:07.4501387Z Ran 1 test in 0.001s 2022-08-17T13:15:07.4501517Z 2022-08-17T13:15:07.4501631Z OK (skipped=1) 2022-08-17T13:15:07.4501731Z 2022-08-17T13:15:07.4501821Z Generating XML reports... 2022-08-17T13:15:07.4532235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131507.xml 2022-08-17T13:15:08.6330898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:08.6331305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:08.6333040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:08.6333551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:08.7855411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:08.7865326Z 2022-08-17T13:15:08.7865475Z Running tests... 2022-08-17T13:15:08.7866230Z ---------------------------------------------------------------------- 2022-08-17T13:15:08.7871291Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:08.7871698Z 2022-08-17T13:15:08.7872054Z ---------------------------------------------------------------------- 2022-08-17T13:15:08.7872320Z Ran 1 test in 0.001s 2022-08-17T13:15:08.7872440Z 2022-08-17T13:15:08.7872517Z OK (skipped=1) 2022-08-17T13:15:08.7872632Z 2022-08-17T13:15:08.7872709Z Generating XML reports... 2022-08-17T13:15:08.7902129Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131508.xml 2022-08-17T13:15:09.9535666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:09.9536563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:09.9538506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:09.9539110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:10.1072252Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:10.1082410Z 2022-08-17T13:15:10.1082727Z Running tests... 2022-08-17T13:15:10.1083328Z ---------------------------------------------------------------------- 2022-08-17T13:15:10.1089400Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:10.1089809Z 2022-08-17T13:15:10.1090211Z ---------------------------------------------------------------------- 2022-08-17T13:15:10.1090637Z Ran 1 test in 0.001s 2022-08-17T13:15:10.1090821Z 2022-08-17T13:15:10.1090944Z OK (skipped=1) 2022-08-17T13:15:10.1091142Z 2022-08-17T13:15:10.1091293Z Generating XML reports... 2022-08-17T13:15:10.1121391Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131510.xml 2022-08-17T13:15:11.2671969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:11.2672655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:11.2675337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:11.2675959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:11.4207471Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:11.4218318Z 2022-08-17T13:15:11.4218708Z Running tests... 2022-08-17T13:15:11.4219310Z ---------------------------------------------------------------------- 2022-08-17T13:15:11.4223228Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:11.4223696Z 2022-08-17T13:15:11.4224100Z ---------------------------------------------------------------------- 2022-08-17T13:15:11.4224560Z Ran 1 test in 0.001s 2022-08-17T13:15:11.4224769Z 2022-08-17T13:15:11.4224859Z OK (skipped=1) 2022-08-17T13:15:11.4224975Z 2022-08-17T13:15:11.4225064Z Generating XML reports... 2022-08-17T13:15:11.4255183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131511.xml 2022-08-17T13:15:12.5982292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:12.5982691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:12.5984838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:12.5985222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:12.7540312Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:12.7550555Z 2022-08-17T13:15:12.7550859Z Running tests... 2022-08-17T13:15:12.7551542Z ---------------------------------------------------------------------- 2022-08-17T13:15:12.7556193Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:12.7556663Z 2022-08-17T13:15:12.7557082Z ---------------------------------------------------------------------- 2022-08-17T13:15:12.7557428Z Ran 1 test in 0.001s 2022-08-17T13:15:12.7557550Z 2022-08-17T13:15:12.7557625Z OK (skipped=1) 2022-08-17T13:15:12.7557727Z 2022-08-17T13:15:12.7557814Z Generating XML reports... 2022-08-17T13:15:12.7589535Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131512.xml 2022-08-17T13:15:13.9230531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:13.9230922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:13.9233085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:13.9233560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:14.0792288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:14.0801576Z 2022-08-17T13:15:14.0801680Z Running tests... 2022-08-17T13:15:14.0802254Z ---------------------------------------------------------------------- 2022-08-17T13:15:14.0807296Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:14.0807688Z 2022-08-17T13:15:14.0807994Z ---------------------------------------------------------------------- 2022-08-17T13:15:14.0808257Z Ran 1 test in 0.001s 2022-08-17T13:15:14.0808364Z 2022-08-17T13:15:14.0808441Z OK (skipped=1) 2022-08-17T13:15:14.0808554Z 2022-08-17T13:15:14.0808644Z Generating XML reports... 2022-08-17T13:15:14.0838389Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131514.xml 2022-08-17T13:15:15.2404977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:15.2405597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:15.2407322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:15.2407939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:15.3928800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:15.3939457Z 2022-08-17T13:15:15.3939903Z Running tests... 2022-08-17T13:15:15.3940317Z ---------------------------------------------------------------------- 2022-08-17T13:15:15.3945369Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:15.3945830Z 2022-08-17T13:15:15.3946268Z ---------------------------------------------------------------------- 2022-08-17T13:15:15.3946584Z Ran 1 test in 0.001s 2022-08-17T13:15:15.3946703Z 2022-08-17T13:15:15.3946784Z OK (skipped=1) 2022-08-17T13:15:15.3946901Z 2022-08-17T13:15:15.3946990Z Generating XML reports... 2022-08-17T13:15:15.3977127Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131515.xml 2022-08-17T13:15:16.5570314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:16.5570749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:16.5573225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:16.5573611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:16.7102984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:16.7113446Z 2022-08-17T13:15:16.7113546Z Running tests... 2022-08-17T13:15:16.7114124Z ---------------------------------------------------------------------- 2022-08-17T13:15:16.7120122Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:16.7120665Z 2022-08-17T13:15:16.7121039Z ---------------------------------------------------------------------- 2022-08-17T13:15:16.7121305Z Ran 1 test in 0.001s 2022-08-17T13:15:16.7121680Z 2022-08-17T13:15:16.7121755Z OK (skipped=1) 2022-08-17T13:15:16.7121873Z 2022-08-17T13:15:16.7121962Z Generating XML reports... 2022-08-17T13:15:16.7152404Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131516.xml 2022-08-17T13:15:17.9042836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:17.9043216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:17.9045505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:17.9046069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:18.0608360Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:18.0618538Z 2022-08-17T13:15:18.0618651Z Running tests... 2022-08-17T13:15:18.0619096Z ---------------------------------------------------------------------- 2022-08-17T13:15:18.0624991Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:18.0625501Z 2022-08-17T13:15:18.0626511Z ---------------------------------------------------------------------- 2022-08-17T13:15:18.0633629Z Ran 1 test in 0.001s 2022-08-17T13:15:18.0633804Z 2022-08-17T13:15:18.0633890Z OK (skipped=1) 2022-08-17T13:15:18.0633993Z 2022-08-17T13:15:18.0634084Z Generating XML reports... 2022-08-17T13:15:18.0660169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131518.xml 2022-08-17T13:15:19.2550643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:19.2551184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:19.2552386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:19.2552916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:19.4094691Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:19.4104579Z 2022-08-17T13:15:19.4104710Z Running tests... 2022-08-17T13:15:19.4105672Z ---------------------------------------------------------------------- 2022-08-17T13:15:19.4109999Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:19.4111653Z 2022-08-17T13:15:19.4111998Z ---------------------------------------------------------------------- 2022-08-17T13:15:19.4112260Z Ran 1 test in 0.001s 2022-08-17T13:15:19.4112386Z 2022-08-17T13:15:19.4112453Z OK (skipped=1) 2022-08-17T13:15:19.4112566Z 2022-08-17T13:15:19.4112656Z Generating XML reports... 2022-08-17T13:15:19.4142941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131519.xml 2022-08-17T13:15:20.6105308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:20.6105721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:20.6108219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:20.6108594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:20.7669500Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:20.7679857Z 2022-08-17T13:15:20.7680303Z Running tests... 2022-08-17T13:15:20.7680912Z ---------------------------------------------------------------------- 2022-08-17T13:15:20.7685188Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:15:20.7686176Z 2022-08-17T13:15:20.7686610Z ---------------------------------------------------------------------- 2022-08-17T13:15:20.7687074Z Ran 1 test in 0.001s 2022-08-17T13:15:20.7687307Z 2022-08-17T13:15:20.7687456Z OK (skipped=1) 2022-08-17T13:15:20.7687603Z 2022-08-17T13:15:20.7687706Z Generating XML reports... 2022-08-17T13:15:20.7717472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131520.xml 2022-08-17T13:15:21.9706165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:21.9706577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:21.9709064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:21.9709450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:22.1301813Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:22.1311725Z 2022-08-17T13:15:22.1311832Z Running tests... 2022-08-17T13:15:22.1312638Z ---------------------------------------------------------------------- 2022-08-17T13:15:22.1320756Z test_channels_last_contig (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:15:22.1321189Z 2022-08-17T13:15:22.1321649Z ---------------------------------------------------------------------- 2022-08-17T13:15:22.1321961Z Ran 1 test in 0.001s 2022-08-17T13:15:22.1322088Z 2022-08-17T13:15:22.1322167Z OK (skipped=1) 2022-08-17T13:15:22.1322271Z 2022-08-17T13:15:22.1322361Z Generating XML reports... 2022-08-17T13:15:22.1355594Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131522.xml 2022-08-17T13:15:23.3294797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:23.3295216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:23.3297484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:23.3298057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:23.4842005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:23.4852252Z 2022-08-17T13:15:23.4852395Z Running tests... 2022-08-17T13:15:23.4853143Z ---------------------------------------------------------------------- 2022-08-17T13:15:23.4860481Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-08-17T13:15:23.7468608Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4604 2022-08-17T13:15:23.7496888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4605 2022-08-17T13:15:24.7153288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:24.7153937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:24.7156843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:24.7157431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:24.7208419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:24.7209082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:24.7212805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:24.7213475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:24.8714745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:24.8772406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:25.1531336Z skip: Need at least 2 CUDA devices (1.667s) 2022-08-17T13:15:25.1531626Z 2022-08-17T13:15:25.1532172Z ---------------------------------------------------------------------- 2022-08-17T13:15:25.1532476Z Ran 1 test in 1.668s 2022-08-17T13:15:25.1532600Z 2022-08-17T13:15:25.1532665Z OK (skipped=1) 2022-08-17T13:15:25.1532779Z 2022-08-17T13:15:25.1532876Z Generating XML reports... 2022-08-17T13:15:25.1567897Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131523.xml 2022-08-17T13:15:26.4777237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:26.4777653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:26.4779992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:26.4780506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:26.6338200Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:26.6347465Z 2022-08-17T13:15:26.6347598Z Running tests... 2022-08-17T13:15:26.6348216Z ---------------------------------------------------------------------- 2022-08-17T13:15:26.6355891Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:15:26.9008325Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4637 2022-08-17T13:15:26.9035586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4638 2022-08-17T13:15:27.8947776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:27.8948180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:27.8951190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:27.8951610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:27.8981422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:27.8981821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:27.8985284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:27.8985769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:28.0523513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:28.0543644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:28.3070228Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:15:28.3070526Z 2022-08-17T13:15:28.3071089Z ---------------------------------------------------------------------- 2022-08-17T13:15:28.3071411Z Ran 1 test in 1.672s 2022-08-17T13:15:28.3071535Z 2022-08-17T13:15:28.3071611Z OK (skipped=1) 2022-08-17T13:15:28.3071713Z 2022-08-17T13:15:28.3071803Z Generating XML reports... 2022-08-17T13:15:28.3107590Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131526.xml 2022-08-17T13:15:29.6279967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:29.6280393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:29.6282622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:29.6283070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:29.7848764Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:29.7858677Z 2022-08-17T13:15:29.7858779Z Running tests... 2022-08-17T13:15:29.7859377Z ---------------------------------------------------------------------- 2022-08-17T13:15:29.7869318Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:15:30.0531980Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4670 2022-08-17T13:15:30.0559299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4671 2022-08-17T13:15:31.0750530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:31.0750979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:31.0753500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:31.0754474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:31.0879072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:31.0879582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:31.0882461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:31.0882840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:31.2310980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:31.2438308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:31.5594951Z skip: Need at least 2 CUDA devices (1.773s) 2022-08-17T13:15:31.5595239Z 2022-08-17T13:15:31.5595700Z ---------------------------------------------------------------------- 2022-08-17T13:15:31.5596120Z Ran 1 test in 1.773s 2022-08-17T13:15:31.5596319Z 2022-08-17T13:15:31.5596419Z OK (skipped=1) 2022-08-17T13:15:31.5596596Z 2022-08-17T13:15:31.5596753Z Generating XML reports... 2022-08-17T13:15:31.5632828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131529.xml 2022-08-17T13:15:32.8636768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:32.8637186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:32.8639374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:32.8639775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:33.0177501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:33.0186599Z 2022-08-17T13:15:33.0186754Z Running tests... 2022-08-17T13:15:33.0187185Z ---------------------------------------------------------------------- 2022-08-17T13:15:33.0197341Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:15:33.2792364Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4703 2022-08-17T13:15:33.2820461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4704 2022-08-17T13:15:34.2592555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:34.2592968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:34.2595841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:34.2596223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:34.2840050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:34.2840419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:34.2843676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:34.2844100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:34.4165502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:34.4378709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:34.6856495Z skip: Need at least 2 CUDA devices (1.667s) 2022-08-17T13:15:34.6856783Z 2022-08-17T13:15:34.6857265Z ---------------------------------------------------------------------- 2022-08-17T13:15:34.6857672Z Ran 1 test in 1.667s 2022-08-17T13:15:34.6857877Z 2022-08-17T13:15:34.6858000Z OK (skipped=1) 2022-08-17T13:15:34.6858187Z 2022-08-17T13:15:34.6858577Z Generating XML reports... 2022-08-17T13:15:34.6894731Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131533.xml 2022-08-17T13:15:35.9805494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:35.9806038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:35.9808244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:35.9808624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:36.1325318Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:36.1335336Z 2022-08-17T13:15:36.1335487Z Running tests... 2022-08-17T13:15:36.1335899Z ---------------------------------------------------------------------- 2022-08-17T13:15:36.1344255Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:15:36.3946483Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4736 2022-08-17T13:15:36.3973337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4737 2022-08-17T13:15:37.3749578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:37.3749995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:37.3752714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:37.3753456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:37.3754318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:37.3754964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:37.3757704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:37.3758116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:37.5311304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:37.5322650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:37.8011928Z skip: Need at least 2 CUDA devices (1.667s) 2022-08-17T13:15:37.8012183Z 2022-08-17T13:15:37.8012846Z ---------------------------------------------------------------------- 2022-08-17T13:15:37.8013127Z Ran 1 test in 1.668s 2022-08-17T13:15:37.8013248Z 2022-08-17T13:15:37.8013325Z OK (skipped=1) 2022-08-17T13:15:37.8013503Z 2022-08-17T13:15:37.8013597Z Generating XML reports... 2022-08-17T13:15:37.8048396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131536.xml 2022-08-17T13:15:39.1227908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:39.1228535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:39.1230893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:39.1231482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:39.2774849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:39.2785490Z 2022-08-17T13:15:39.2785810Z Running tests... 2022-08-17T13:15:39.2786419Z ---------------------------------------------------------------------- 2022-08-17T13:15:39.2794718Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:15:39.5439330Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4769 2022-08-17T13:15:39.5467214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4770 2022-08-17T13:15:40.5315849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:40.5316241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:40.5318568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:40.5318962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:40.5639805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:40.5640177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:40.5643719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:40.5644167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:40.6849239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:40.7164004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:41.0503894Z skip: Need at least 2 CUDA devices (1.771s) 2022-08-17T13:15:41.0504393Z 2022-08-17T13:15:41.0504967Z ---------------------------------------------------------------------- 2022-08-17T13:15:41.0505273Z Ran 1 test in 1.772s 2022-08-17T13:15:41.0505383Z 2022-08-17T13:15:41.0505471Z OK (skipped=1) 2022-08-17T13:15:41.0505583Z 2022-08-17T13:15:41.0505671Z Generating XML reports... 2022-08-17T13:15:41.0544831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131539.xml 2022-08-17T13:15:42.3662349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:42.3663145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:42.3665737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:42.3666348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:42.5224152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:42.5234570Z 2022-08-17T13:15:42.5234774Z Running tests... 2022-08-17T13:15:42.5235619Z ---------------------------------------------------------------------- 2022-08-17T13:15:42.5246569Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:15:42.7895624Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4802 2022-08-17T13:15:42.7926042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4803 2022-08-17T13:15:43.7747481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:43.7747883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:43.7750356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:43.7750731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:43.7768912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:43.7769286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:43.7772942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:43.7773328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:43.9338478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:43.9428424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:44.1958665Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:15:44.1959009Z 2022-08-17T13:15:44.1959469Z ---------------------------------------------------------------------- 2022-08-17T13:15:44.1959735Z Ran 1 test in 1.672s 2022-08-17T13:15:44.1959855Z 2022-08-17T13:15:44.1959933Z OK (skipped=1) 2022-08-17T13:15:44.1960062Z 2022-08-17T13:15:44.1960151Z Generating XML reports... 2022-08-17T13:15:44.1995944Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131542.xml 2022-08-17T13:15:45.4857001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:45.4857616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:45.4859702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:45.4860295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:45.6374434Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:45.6384623Z 2022-08-17T13:15:45.6385184Z Running tests... 2022-08-17T13:15:45.6385807Z ---------------------------------------------------------------------- 2022-08-17T13:15:45.6395673Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:15:45.9032100Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4835 2022-08-17T13:15:45.9059542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4836 2022-08-17T13:15:46.8775081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:46.8775619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:46.8776441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:46.8776949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:46.8777957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:46.8778648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:46.8780896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:46.8781427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:47.0340741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:47.0342826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:47.3094156Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T13:15:47.3094352Z 2022-08-17T13:15:47.3094748Z ---------------------------------------------------------------------- 2022-08-17T13:15:47.3095012Z Ran 1 test in 1.671s 2022-08-17T13:15:47.3095120Z 2022-08-17T13:15:47.3095199Z OK (skipped=1) 2022-08-17T13:15:47.3095335Z 2022-08-17T13:15:47.3095488Z Generating XML reports... 2022-08-17T13:15:47.3130681Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131545.xml 2022-08-17T13:15:48.6050966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:48.6051942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:48.6054349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:48.6054939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:48.7618292Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:48.7628817Z 2022-08-17T13:15:48.7629102Z Running tests... 2022-08-17T13:15:48.7629718Z ---------------------------------------------------------------------- 2022-08-17T13:15:48.7637558Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-08-17T13:15:49.0333974Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4868 2022-08-17T13:15:49.0365086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4869 2022-08-17T13:15:50.0131794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:50.0132460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:50.0134847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:50.0135494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:50.0156441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:50.0157110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:50.0161064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:50.0161478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:50.1751548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:50.1805244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:50.4399010Z skip: Need at least 2 CUDA devices (1.677s) 2022-08-17T13:15:50.4399285Z 2022-08-17T13:15:50.4399712Z ---------------------------------------------------------------------- 2022-08-17T13:15:50.4400053Z Ran 1 test in 1.677s 2022-08-17T13:15:50.4400287Z 2022-08-17T13:15:50.4400398Z OK (skipped=1) 2022-08-17T13:15:50.4400615Z 2022-08-17T13:15:50.4402497Z Generating XML reports... 2022-08-17T13:15:50.4436483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131548.xml 2022-08-17T13:15:51.7318625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:51.7319017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:51.7321482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:51.7321849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:51.8851722Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:51.8860979Z 2022-08-17T13:15:51.8861112Z Running tests... 2022-08-17T13:15:51.8861658Z ---------------------------------------------------------------------- 2022-08-17T13:15:51.8872439Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:15:52.1482724Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4901 2022-08-17T13:15:52.1511289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4902 2022-08-17T13:15:53.1100213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:53.1100615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:53.1102588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:53.1103189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:53.1466844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:53.1467356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:53.1470318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:53.1470839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:53.2648270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:53.3007090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:53.5548615Z skip: Need at least 2 CUDA devices (1.668s) 2022-08-17T13:15:53.5548918Z 2022-08-17T13:15:53.5549303Z ---------------------------------------------------------------------- 2022-08-17T13:15:53.5549765Z Ran 1 test in 1.669s 2022-08-17T13:15:53.5549966Z 2022-08-17T13:15:53.5550101Z OK (skipped=1) 2022-08-17T13:15:53.5550285Z 2022-08-17T13:15:53.5550379Z Generating XML reports... 2022-08-17T13:15:53.5586324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131551.xml 2022-08-17T13:15:54.8700721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:54.8701426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:54.8704539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:54.8705124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:55.0267519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:55.0278310Z 2022-08-17T13:15:55.0278781Z Running tests... 2022-08-17T13:15:55.0279203Z ---------------------------------------------------------------------- 2022-08-17T13:15:55.0289866Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:15:55.2933444Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4934 2022-08-17T13:15:55.2962821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4935 2022-08-17T13:15:56.2722638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:56.2723370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:56.2724061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:56.2724663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:56.2725450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:56.2726291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:56.2727044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:56.2727673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:56.4304720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:56.4327514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:56.6998741Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:15:56.6998958Z 2022-08-17T13:15:56.6999273Z ---------------------------------------------------------------------- 2022-08-17T13:15:56.6999538Z Ran 1 test in 1.672s 2022-08-17T13:15:56.6999659Z 2022-08-17T13:15:56.6999741Z OK (skipped=1) 2022-08-17T13:15:56.6999855Z 2022-08-17T13:15:56.6999946Z Generating XML reports... 2022-08-17T13:15:56.7035425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131555.xml 2022-08-17T13:15:58.0133640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:58.0134290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:58.0136785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:58.0137371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:58.1662625Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:15:58.1672766Z 2022-08-17T13:15:58.1673221Z Running tests... 2022-08-17T13:15:58.1673653Z ---------------------------------------------------------------------- 2022-08-17T13:15:58.1684990Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-08-17T13:15:58.4326163Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4967 2022-08-17T13:15:58.4353404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4968 2022-08-17T13:15:59.4139039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:59.4139434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:59.4144284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:59.4144700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:59.4436417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:15:59.4437097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:15:59.4439878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:15:59.4440495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:15:59.5770773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:15:59.5991998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:15:59.8391522Z skip: Need at least 2 CUDA devices (1.671s) 2022-08-17T13:15:59.8391950Z 2022-08-17T13:15:59.8392494Z ---------------------------------------------------------------------- 2022-08-17T13:15:59.8392754Z Ran 1 test in 1.672s 2022-08-17T13:15:59.8392875Z 2022-08-17T13:15:59.8392952Z OK (skipped=1) 2022-08-17T13:15:59.8393067Z 2022-08-17T13:15:59.8393155Z Generating XML reports... 2022-08-17T13:15:59.8428426Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131558.xml 2022-08-17T13:16:01.1376676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:01.1377066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:01.1380064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:01.1380607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:01.2957774Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:01.2967702Z 2022-08-17T13:16:01.2967858Z Running tests... 2022-08-17T13:16:01.2968259Z ---------------------------------------------------------------------- 2022-08-17T13:16:01.2980101Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-08-17T13:16:01.5646081Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5000 2022-08-17T13:16:01.5674153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5001 2022-08-17T13:16:02.5302504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:02.5302894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:02.5306183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:02.5306604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:02.5475886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:02.5476257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:02.5478982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:02.5479363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:02.6839407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:16:02.7006395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:16:02.9708568Z skip: Need at least 2 CUDA devices (1.674s) 2022-08-17T13:16:02.9709654Z 2022-08-17T13:16:02.9709982Z ---------------------------------------------------------------------- 2022-08-17T13:16:02.9710258Z Ran 1 test in 1.674s 2022-08-17T13:16:02.9710381Z 2022-08-17T13:16:02.9710445Z OK (skipped=1) 2022-08-17T13:16:02.9710555Z 2022-08-17T13:16:02.9710648Z Generating XML reports... 2022-08-17T13:16:02.9746624Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131601.xml 2022-08-17T13:16:04.2905740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:04.2906409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:04.2909150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:04.2910087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:04.4428818Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:04.4439005Z 2022-08-17T13:16:04.4439288Z Running tests... 2022-08-17T13:16:04.4439928Z ---------------------------------------------------------------------- 2022-08-17T13:16:04.4445278Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:04.4445742Z 2022-08-17T13:16:04.4446387Z ---------------------------------------------------------------------- 2022-08-17T13:16:04.4446664Z Ran 1 test in 0.001s 2022-08-17T13:16:04.4446784Z 2022-08-17T13:16:04.4446859Z OK (skipped=1) 2022-08-17T13:16:04.4446971Z 2022-08-17T13:16:04.4447060Z Generating XML reports... 2022-08-17T13:16:04.4478785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131604.xml 2022-08-17T13:16:05.5999980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:05.6000385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:05.6002929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:05.6003299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:05.7537996Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:05.7548387Z 2022-08-17T13:16:05.7548516Z Running tests... 2022-08-17T13:16:05.7549118Z ---------------------------------------------------------------------- 2022-08-17T13:16:05.7553924Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:05.7554388Z 2022-08-17T13:16:05.7554791Z ---------------------------------------------------------------------- 2022-08-17T13:16:05.7555240Z Ran 1 test in 0.001s 2022-08-17T13:16:05.7555371Z 2022-08-17T13:16:05.7555447Z OK (skipped=1) 2022-08-17T13:16:05.7555562Z 2022-08-17T13:16:05.7555644Z Generating XML reports... 2022-08-17T13:16:05.7584429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131605.xml 2022-08-17T13:16:06.9260690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:06.9261124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:06.9263456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:06.9263839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:07.0822080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:07.0832248Z 2022-08-17T13:16:07.0832373Z Running tests... 2022-08-17T13:16:07.0832994Z ---------------------------------------------------------------------- 2022-08-17T13:16:07.0837955Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:07.0838427Z 2022-08-17T13:16:07.0838871Z ---------------------------------------------------------------------- 2022-08-17T13:16:07.0839169Z Ran 1 test in 0.001s 2022-08-17T13:16:07.0839289Z 2022-08-17T13:16:07.0839365Z OK (skipped=1) 2022-08-17T13:16:07.0839464Z 2022-08-17T13:16:07.0839554Z Generating XML reports... 2022-08-17T13:16:07.0868148Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131607.xml 2022-08-17T13:16:08.2437855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:08.2438512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:08.2440474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:08.2440848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:08.3986978Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:08.3996652Z 2022-08-17T13:16:08.3996752Z Running tests... 2022-08-17T13:16:08.3997163Z ---------------------------------------------------------------------- 2022-08-17T13:16:08.4009924Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:16:08.4011109Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:08.4011689Z 2022-08-17T13:16:08.4012182Z ---------------------------------------------------------------------- 2022-08-17T13:16:08.4012436Z Ran 1 test in 0.001s 2022-08-17T13:16:08.4012557Z 2022-08-17T13:16:08.4012634Z OK (skipped=1) 2022-08-17T13:16:08.4012748Z 2022-08-17T13:16:08.4012873Z Generating XML reports... 2022-08-17T13:16:08.4042218Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131608.xml 2022-08-17T13:16:09.5745169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:09.5745552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:09.5747536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:09.5747917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:09.7314580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:09.7324643Z 2022-08-17T13:16:09.7324726Z Running tests... 2022-08-17T13:16:09.7325379Z ---------------------------------------------------------------------- 2022-08-17T13:16:09.7333681Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-08-17T13:16:09.7334833Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:09.7335501Z 2022-08-17T13:16:09.7335932Z ---------------------------------------------------------------------- 2022-08-17T13:16:09.7336358Z Ran 1 test in 0.001s 2022-08-17T13:16:09.7336552Z 2022-08-17T13:16:09.7336663Z OK (skipped=1) 2022-08-17T13:16:09.7336848Z 2022-08-17T13:16:09.7336999Z Generating XML reports... 2022-08-17T13:16:09.7370103Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131609.xml 2022-08-17T13:16:10.9087433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:10.9088036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:10.9090436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:10.9090842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:11.0609327Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:11.0618809Z 2022-08-17T13:16:11.0618905Z Running tests... 2022-08-17T13:16:11.0619521Z ---------------------------------------------------------------------- 2022-08-17T13:16:11.0633114Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:11.0633605Z 2022-08-17T13:16:11.0633962Z ---------------------------------------------------------------------- 2022-08-17T13:16:11.0634423Z Ran 1 test in 0.001s 2022-08-17T13:16:11.0634548Z 2022-08-17T13:16:11.0634624Z OK (skipped=1) 2022-08-17T13:16:11.0634739Z 2022-08-17T13:16:11.0634815Z Generating XML reports... 2022-08-17T13:16:11.0664860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131611.xml 2022-08-17T13:16:12.2226062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:12.2226469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:12.2228827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:12.2229207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:12.3738015Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:12.3747941Z 2022-08-17T13:16:12.3748267Z Running tests... 2022-08-17T13:16:12.3748664Z ---------------------------------------------------------------------- 2022-08-17T13:16:12.3763503Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:12.3764164Z 2022-08-17T13:16:12.3764436Z ---------------------------------------------------------------------- 2022-08-17T13:16:12.3764766Z Ran 1 test in 0.002s 2022-08-17T13:16:12.3764887Z 2022-08-17T13:16:12.3764963Z OK (skipped=1) 2022-08-17T13:16:12.3765077Z 2022-08-17T13:16:12.3765166Z Generating XML reports... 2022-08-17T13:16:12.3794249Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131612.xml 2022-08-17T13:16:13.5424307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:13.5424992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:13.5450776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:13.5451306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:13.6955406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:13.6964539Z 2022-08-17T13:16:13.6964696Z Running tests... 2022-08-17T13:16:13.6965349Z ---------------------------------------------------------------------- 2022-08-17T13:16:13.6973009Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:13.6973518Z 2022-08-17T13:16:13.6973905Z ---------------------------------------------------------------------- 2022-08-17T13:16:13.6974153Z Ran 1 test in 0.001s 2022-08-17T13:16:13.6974273Z 2022-08-17T13:16:13.6974351Z OK (skipped=1) 2022-08-17T13:16:13.6974481Z 2022-08-17T13:16:13.6974571Z Generating XML reports... 2022-08-17T13:16:13.7004700Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131613.xml 2022-08-17T13:16:14.8721852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:14.8722244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:14.8724500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:14.8724864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:15.0266812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:15.0276731Z 2022-08-17T13:16:15.0276835Z Running tests... 2022-08-17T13:16:15.0277286Z ---------------------------------------------------------------------- 2022-08-17T13:16:15.0282388Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:15.0282724Z 2022-08-17T13:16:15.0283166Z ---------------------------------------------------------------------- 2022-08-17T13:16:15.0283644Z Ran 1 test in 0.001s 2022-08-17T13:16:15.0283836Z 2022-08-17T13:16:15.0283916Z OK (skipped=1) 2022-08-17T13:16:15.0284030Z 2022-08-17T13:16:15.0284120Z Generating XML reports... 2022-08-17T13:16:15.0315320Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131615.xml 2022-08-17T13:16:16.2225612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:16.2226220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:16.2228667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:16.2229277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:16.3805226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:16.3816357Z 2022-08-17T13:16:16.3816715Z Running tests... 2022-08-17T13:16:16.3817630Z ---------------------------------------------------------------------- 2022-08-17T13:16:16.3822125Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:16.3822709Z 2022-08-17T13:16:16.3823128Z ---------------------------------------------------------------------- 2022-08-17T13:16:16.3823620Z Ran 1 test in 0.001s 2022-08-17T13:16:16.3823813Z 2022-08-17T13:16:16.3823945Z OK (skipped=1) 2022-08-17T13:16:16.3824123Z 2022-08-17T13:16:16.3824265Z Generating XML reports... 2022-08-17T13:16:16.3857351Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131616.xml 2022-08-17T13:16:17.5693922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:17.5694506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:17.5696204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:17.5696842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:17.7243118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:17.7252912Z 2022-08-17T13:16:17.7253175Z Running tests... 2022-08-17T13:16:17.7253938Z ---------------------------------------------------------------------- 2022-08-17T13:16:17.7275874Z test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:16:17.7276392Z 2022-08-17T13:16:17.7276872Z ---------------------------------------------------------------------- 2022-08-17T13:16:17.7277339Z Ran 1 test in 0.002s 2022-08-17T13:16:17.7277495Z 2022-08-17T13:16:17.7277569Z OK (skipped=1) 2022-08-17T13:16:17.7277684Z 2022-08-17T13:16:17.7277782Z Generating XML reports... 2022-08-17T13:16:17.7309603Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131617.xml 2022-08-17T13:16:18.9139500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:18.9139901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:18.9142209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:18.9142564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:19.0724582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:19.0734485Z 2022-08-17T13:16:19.0735069Z Running tests... 2022-08-17T13:16:19.0735522Z ---------------------------------------------------------------------- 2022-08-17T13:16:19.0740671Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:19.0740998Z 2022-08-17T13:16:19.0741303Z ---------------------------------------------------------------------- 2022-08-17T13:16:19.0741551Z Ran 1 test in 0.001s 2022-08-17T13:16:19.0741670Z 2022-08-17T13:16:19.0741747Z OK (skipped=1) 2022-08-17T13:16:19.0741864Z 2022-08-17T13:16:19.0741998Z Generating XML reports... 2022-08-17T13:16:19.0771430Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131619.xml 2022-08-17T13:16:20.2299048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:20.2299460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:20.2302713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:20.2303120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:20.3871696Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:20.3880974Z 2022-08-17T13:16:20.3881298Z Running tests... 2022-08-17T13:16:20.3881903Z ---------------------------------------------------------------------- 2022-08-17T13:16:20.3886984Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:20.3887443Z 2022-08-17T13:16:20.3887814Z ---------------------------------------------------------------------- 2022-08-17T13:16:20.3888078Z Ran 1 test in 0.001s 2022-08-17T13:16:20.3888195Z 2022-08-17T13:16:20.3888273Z OK (skipped=1) 2022-08-17T13:16:20.3888386Z 2022-08-17T13:16:20.3888475Z Generating XML reports... 2022-08-17T13:16:20.3917519Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131620.xml 2022-08-17T13:16:21.5609512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:21.5609901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:21.5612378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:21.5612786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:21.7171640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:21.7182023Z 2022-08-17T13:16:21.7182113Z Running tests... 2022-08-17T13:16:21.7182704Z ---------------------------------------------------------------------- 2022-08-17T13:16:21.7188593Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:21.7189140Z 2022-08-17T13:16:21.7189436Z ---------------------------------------------------------------------- 2022-08-17T13:16:21.7189686Z Ran 1 test in 0.001s 2022-08-17T13:16:21.7189809Z 2022-08-17T13:16:21.7189886Z OK (skipped=1) 2022-08-17T13:16:21.7190001Z 2022-08-17T13:16:21.7190095Z Generating XML reports... 2022-08-17T13:16:21.7226577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131621.xml 2022-08-17T13:16:22.9028290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:22.9028739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:22.9030981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:22.9031607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:23.0580232Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:23.0590287Z 2022-08-17T13:16:23.0590417Z Running tests... 2022-08-17T13:16:23.0590989Z ---------------------------------------------------------------------- 2022-08-17T13:16:23.0596738Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:23.0597303Z 2022-08-17T13:16:23.0597657Z ---------------------------------------------------------------------- 2022-08-17T13:16:23.0597921Z Ran 1 test in 0.001s 2022-08-17T13:16:23.0598026Z 2022-08-17T13:16:23.0598102Z OK (skipped=1) 2022-08-17T13:16:23.0598216Z 2022-08-17T13:16:23.0598306Z Generating XML reports... 2022-08-17T13:16:23.0628893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131623.xml 2022-08-17T13:16:24.2282817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:24.2283441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:24.2285487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:24.2286155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:24.3831605Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:24.3841948Z 2022-08-17T13:16:24.3842051Z Running tests... 2022-08-17T13:16:24.3842656Z ---------------------------------------------------------------------- 2022-08-17T13:16:24.3849573Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:24.3850173Z 2022-08-17T13:16:24.3850490Z ---------------------------------------------------------------------- 2022-08-17T13:16:24.3850742Z Ran 1 test in 0.001s 2022-08-17T13:16:24.3850862Z 2022-08-17T13:16:24.3850940Z OK (skipped=1) 2022-08-17T13:16:24.3851063Z 2022-08-17T13:16:24.3851152Z Generating XML reports... 2022-08-17T13:16:24.3880886Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131624.xml 2022-08-17T13:16:25.5470459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:25.5470901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:25.5472860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:25.5473240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:25.7013651Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:25.7023691Z 2022-08-17T13:16:25.7023822Z Running tests... 2022-08-17T13:16:25.7024327Z ---------------------------------------------------------------------- 2022-08-17T13:16:25.7030009Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:25.7030491Z 2022-08-17T13:16:25.7030924Z ---------------------------------------------------------------------- 2022-08-17T13:16:25.7031304Z Ran 1 test in 0.001s 2022-08-17T13:16:25.7031413Z 2022-08-17T13:16:25.7031491Z OK (skipped=1) 2022-08-17T13:16:25.7031606Z 2022-08-17T13:16:25.7031697Z Generating XML reports... 2022-08-17T13:16:25.7061290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131625.xml 2022-08-17T13:16:26.8730773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:26.8731163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:26.8733303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:26.8733669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:27.0273881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:27.0284037Z 2022-08-17T13:16:27.0284117Z Running tests... 2022-08-17T13:16:27.0284531Z ---------------------------------------------------------------------- 2022-08-17T13:16:27.0290157Z test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:27.0290553Z 2022-08-17T13:16:27.0290834Z ---------------------------------------------------------------------- 2022-08-17T13:16:27.0291162Z Ran 1 test in 0.001s 2022-08-17T13:16:27.0291323Z 2022-08-17T13:16:27.0291401Z OK (skipped=1) 2022-08-17T13:16:27.0291521Z 2022-08-17T13:16:27.0291611Z Generating XML reports... 2022-08-17T13:16:27.0322255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131627.xml 2022-08-17T13:16:28.2070660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:28.2071040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:28.2073535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:28.2073914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:28.3611800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:28.3621735Z 2022-08-17T13:16:28.3621825Z Running tests... 2022-08-17T13:16:28.3622747Z ---------------------------------------------------------------------- 2022-08-17T13:16:28.3628746Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:28.3629214Z 2022-08-17T13:16:28.3629616Z ---------------------------------------------------------------------- 2022-08-17T13:16:28.3630036Z Ran 1 test in 0.001s 2022-08-17T13:16:28.3630230Z 2022-08-17T13:16:28.3630364Z OK (skipped=1) 2022-08-17T13:16:28.3630564Z 2022-08-17T13:16:28.3630724Z Generating XML reports... 2022-08-17T13:16:28.3661133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131628.xml 2022-08-17T13:16:29.5448399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:29.5449039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:29.5451390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:29.7022293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:29.7023257Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:29.7033630Z 2022-08-17T13:16:29.7034067Z Running tests... 2022-08-17T13:16:29.7034492Z ---------------------------------------------------------------------- 2022-08-17T13:16:29.7039390Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:29.7040537Z 2022-08-17T13:16:29.7040776Z ---------------------------------------------------------------------- 2022-08-17T13:16:29.7041038Z Ran 1 test in 0.001s 2022-08-17T13:16:29.7041144Z 2022-08-17T13:16:29.7041221Z OK (skipped=1) 2022-08-17T13:16:29.7041583Z 2022-08-17T13:16:29.7041674Z Generating XML reports... 2022-08-17T13:16:29.7072220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131629.xml 2022-08-17T13:16:30.8951414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:30.8951843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:30.8953992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:30.8954501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:31.0528008Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:31.0537336Z 2022-08-17T13:16:31.0537465Z Running tests... 2022-08-17T13:16:31.0538103Z ---------------------------------------------------------------------- 2022-08-17T13:16:31.0543484Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:31.0543792Z 2022-08-17T13:16:31.0544162Z ---------------------------------------------------------------------- 2022-08-17T13:16:31.0544415Z Ran 1 test in 0.001s 2022-08-17T13:16:31.0544624Z 2022-08-17T13:16:31.0544921Z OK (skipped=1) 2022-08-17T13:16:31.0545041Z 2022-08-17T13:16:31.0545130Z Generating XML reports... 2022-08-17T13:16:31.0577029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131631.xml 2022-08-17T13:16:32.2518125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:32.2518507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:32.2521015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:32.2521408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:32.4076054Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:32.4085302Z 2022-08-17T13:16:32.4085448Z Running tests... 2022-08-17T13:16:32.4086287Z ---------------------------------------------------------------------- 2022-08-17T13:16:32.4093132Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:32.4093712Z 2022-08-17T13:16:32.4094057Z ---------------------------------------------------------------------- 2022-08-17T13:16:32.4094319Z Ran 1 test in 0.001s 2022-08-17T13:16:32.4094442Z 2022-08-17T13:16:32.4094517Z OK (skipped=1) 2022-08-17T13:16:32.4094652Z 2022-08-17T13:16:32.4095061Z Generating XML reports... 2022-08-17T13:16:32.4123684Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131632.xml 2022-08-17T13:16:33.5879370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:33.5879747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:33.5882569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:33.5882974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:33.7409487Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:33.7419001Z 2022-08-17T13:16:33.7419149Z Running tests... 2022-08-17T13:16:33.7419771Z ---------------------------------------------------------------------- 2022-08-17T13:16:33.7428108Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:33.7428964Z 2022-08-17T13:16:33.7429424Z ---------------------------------------------------------------------- 2022-08-17T13:16:33.7429860Z Ran 1 test in 0.001s 2022-08-17T13:16:33.7430070Z 2022-08-17T13:16:33.7430209Z OK (skipped=1) 2022-08-17T13:16:33.7430388Z 2022-08-17T13:16:33.7430479Z Generating XML reports... 2022-08-17T13:16:33.7459780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131633.xml 2022-08-17T13:16:34.9122566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:34.9123199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:34.9129097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:34.9129714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:35.0656547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:35.0665901Z 2022-08-17T13:16:35.0666042Z Running tests... 2022-08-17T13:16:35.0666938Z ---------------------------------------------------------------------- 2022-08-17T13:16:35.3291806Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5263 2022-08-17T13:16:35.3320858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5264 2022-08-17T13:16:36.3150773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:36.3151447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:36.3154470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:36.3155048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:36.3162197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:36.3162856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:36.3166952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:36.3167559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:36.4723439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:16:36.4727960Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4729453Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4730599Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4731592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4732679Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4733534Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4734135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:16:36.4740800Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4742614Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4743903Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4744779Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4745638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.4746486Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:16:36.7355797Z ok (1.668s) 2022-08-17T13:16:36.7355986Z 2022-08-17T13:16:36.7356394Z ---------------------------------------------------------------------- 2022-08-17T13:16:36.7356662Z Ran 1 test in 1.669s 2022-08-17T13:16:36.7356783Z 2022-08-17T13:16:36.7356847Z OK 2022-08-17T13:16:36.7356963Z 2022-08-17T13:16:36.7357124Z Generating XML reports... 2022-08-17T13:16:36.7392942Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131635.xml 2022-08-17T13:16:38.0315963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:38.0316452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:38.0318419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:38.0318778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:38.1881544Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:38.1891890Z 2022-08-17T13:16:38.1892028Z Running tests... 2022-08-17T13:16:38.1892451Z ---------------------------------------------------------------------- 2022-08-17T13:16:38.1898109Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:38.1898519Z 2022-08-17T13:16:38.1898812Z ---------------------------------------------------------------------- 2022-08-17T13:16:38.1899136Z Ran 1 test in 0.001s 2022-08-17T13:16:38.1899256Z 2022-08-17T13:16:38.1899332Z OK (skipped=1) 2022-08-17T13:16:38.1899443Z 2022-08-17T13:16:38.1899519Z Generating XML reports... 2022-08-17T13:16:38.1929840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131638.xml 2022-08-17T13:16:39.3664294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:39.3664967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:39.3666742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:39.3667621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:39.5227388Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:39.5238021Z 2022-08-17T13:16:39.5238263Z Running tests... 2022-08-17T13:16:39.5238630Z ---------------------------------------------------------------------- 2022-08-17T13:16:39.5243978Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:39.5244338Z 2022-08-17T13:16:39.5244585Z ---------------------------------------------------------------------- 2022-08-17T13:16:39.5244854Z Ran 1 test in 0.001s 2022-08-17T13:16:39.5244973Z 2022-08-17T13:16:39.5245050Z OK (skipped=1) 2022-08-17T13:16:39.5245163Z 2022-08-17T13:16:39.5245238Z Generating XML reports... 2022-08-17T13:16:39.5277961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131639.xml 2022-08-17T13:16:40.7067767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:40.7068163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:40.7077099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:40.7077569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:40.8608149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:40.8619272Z 2022-08-17T13:16:40.8619640Z Running tests... 2022-08-17T13:16:40.8620033Z ---------------------------------------------------------------------- 2022-08-17T13:16:40.8626957Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:40.8627430Z 2022-08-17T13:16:40.8627886Z ---------------------------------------------------------------------- 2022-08-17T13:16:40.8628220Z Ran 1 test in 0.001s 2022-08-17T13:16:40.8628342Z 2022-08-17T13:16:40.8628404Z OK (skipped=1) 2022-08-17T13:16:40.8628518Z 2022-08-17T13:16:40.8628609Z Generating XML reports... 2022-08-17T13:16:40.8657359Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131640.xml 2022-08-17T13:16:42.0340503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:42.0340880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:42.0343271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:42.0343735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:42.1880849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:42.1890353Z 2022-08-17T13:16:42.1890802Z Running tests... 2022-08-17T13:16:42.1891425Z ---------------------------------------------------------------------- 2022-08-17T13:16:42.1897365Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:42.1897852Z 2022-08-17T13:16:42.1898233Z ---------------------------------------------------------------------- 2022-08-17T13:16:42.1898683Z Ran 1 test in 0.001s 2022-08-17T13:16:42.1898854Z 2022-08-17T13:16:42.1898934Z OK (skipped=1) 2022-08-17T13:16:42.1899061Z 2022-08-17T13:16:42.1899154Z Generating XML reports... 2022-08-17T13:16:42.1929168Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131642.xml 2022-08-17T13:16:43.3633375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:43.3634035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:43.3636588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:43.3637174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:43.5177143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:43.5187424Z 2022-08-17T13:16:43.5187685Z Running tests... 2022-08-17T13:16:43.5188314Z ---------------------------------------------------------------------- 2022-08-17T13:16:43.5195277Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:43.5195686Z 2022-08-17T13:16:43.5195983Z ---------------------------------------------------------------------- 2022-08-17T13:16:43.5196230Z Ran 1 test in 0.001s 2022-08-17T13:16:43.5196363Z 2022-08-17T13:16:43.5196439Z OK (skipped=1) 2022-08-17T13:16:43.5196553Z 2022-08-17T13:16:43.5196642Z Generating XML reports... 2022-08-17T13:16:43.5226857Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131643.xml 2022-08-17T13:16:44.6839248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:44.6839710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:44.6841823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:44.6842282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:44.8380886Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:44.8390969Z 2022-08-17T13:16:44.8391085Z Running tests... 2022-08-17T13:16:44.8391843Z ---------------------------------------------------------------------- 2022-08-17T13:16:44.8398627Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:44.8399145Z 2022-08-17T13:16:44.8399425Z ---------------------------------------------------------------------- 2022-08-17T13:16:44.8399687Z Ran 1 test in 0.001s 2022-08-17T13:16:44.8399807Z 2022-08-17T13:16:44.8399889Z OK (skipped=1) 2022-08-17T13:16:44.8400002Z 2022-08-17T13:16:44.8400094Z Generating XML reports... 2022-08-17T13:16:44.8432202Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131644.xml 2022-08-17T13:16:46.0071382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:46.0071806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:46.0074344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:46.0074727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:46.1612597Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:46.1622356Z 2022-08-17T13:16:46.1622602Z Running tests... 2022-08-17T13:16:46.1623242Z ---------------------------------------------------------------------- 2022-08-17T13:16:46.1630340Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:46.1630841Z 2022-08-17T13:16:46.1631257Z ---------------------------------------------------------------------- 2022-08-17T13:16:46.1631517Z Ran 1 test in 0.001s 2022-08-17T13:16:46.1631636Z 2022-08-17T13:16:46.1631714Z OK (skipped=1) 2022-08-17T13:16:46.1631814Z 2022-08-17T13:16:46.1631904Z Generating XML reports... 2022-08-17T13:16:46.1662647Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131646.xml 2022-08-17T13:16:47.3452499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:47.3452951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:47.3454616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:47.3455154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:47.4966852Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:47.4976799Z 2022-08-17T13:16:47.4976946Z Running tests... 2022-08-17T13:16:47.4977583Z ---------------------------------------------------------------------- 2022-08-17T13:16:47.4983664Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:47.4984435Z 2022-08-17T13:16:47.4984724Z ---------------------------------------------------------------------- 2022-08-17T13:16:47.4985186Z Ran 1 test in 0.001s 2022-08-17T13:16:47.4985401Z 2022-08-17T13:16:47.4985536Z OK (skipped=1) 2022-08-17T13:16:47.4985740Z 2022-08-17T13:16:47.4985897Z Generating XML reports... 2022-08-17T13:16:47.5015896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131647.xml 2022-08-17T13:16:48.6703906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:48.6704365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:48.6706283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:48.6706797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:48.8290054Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:48.8300289Z 2022-08-17T13:16:48.8300416Z Running tests... 2022-08-17T13:16:48.8301007Z ---------------------------------------------------------------------- 2022-08-17T13:16:48.8306586Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:48.8307112Z 2022-08-17T13:16:48.8307536Z ---------------------------------------------------------------------- 2022-08-17T13:16:48.8307942Z Ran 1 test in 0.001s 2022-08-17T13:16:48.8308432Z 2022-08-17T13:16:48.8308563Z OK (skipped=1) 2022-08-17T13:16:48.8308765Z 2022-08-17T13:16:48.8308924Z Generating XML reports... 2022-08-17T13:16:48.8339636Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131648.xml 2022-08-17T13:16:49.9948303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:49.9948899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:49.9950417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:49.9951044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:50.1492660Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:50.1502856Z 2022-08-17T13:16:50.1503172Z Running tests... 2022-08-17T13:16:50.1503878Z ---------------------------------------------------------------------- 2022-08-17T13:16:50.1510127Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:50.1510639Z 2022-08-17T13:16:50.1511313Z ---------------------------------------------------------------------- 2022-08-17T13:16:50.1511780Z Ran 1 test in 0.001s 2022-08-17T13:16:50.1511994Z 2022-08-17T13:16:50.1512129Z OK (skipped=1) 2022-08-17T13:16:50.1512311Z 2022-08-17T13:16:50.1512476Z Generating XML reports... 2022-08-17T13:16:50.1542283Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131650.xml 2022-08-17T13:16:51.3279232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:51.3279850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:51.3281468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:51.3282117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:51.4832249Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:51.4842899Z 2022-08-17T13:16:51.4843345Z Running tests... 2022-08-17T13:16:51.4843769Z ---------------------------------------------------------------------- 2022-08-17T13:16:51.4858504Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:51.4858960Z 2022-08-17T13:16:51.4859376Z ---------------------------------------------------------------------- 2022-08-17T13:16:51.4859657Z Ran 1 test in 0.002s 2022-08-17T13:16:51.4859777Z 2022-08-17T13:16:51.4859853Z OK (skipped=1) 2022-08-17T13:16:51.4859966Z 2022-08-17T13:16:51.4860043Z Generating XML reports... 2022-08-17T13:16:51.4890534Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131651.xml 2022-08-17T13:16:52.6524136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:52.6524790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:52.6526967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:52.6527574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:52.8062152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:52.8072730Z 2022-08-17T13:16:52.8073201Z Running tests... 2022-08-17T13:16:52.8073633Z ---------------------------------------------------------------------- 2022-08-17T13:16:52.8091081Z test_no_grad (__main__.DistributedDataParallelTest) 2022-08-17T13:16:52.8092034Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:16:52.8092442Z 2022-08-17T13:16:52.8092831Z ---------------------------------------------------------------------- 2022-08-17T13:16:52.8093252Z Ran 1 test in 0.002s 2022-08-17T13:16:52.8093464Z 2022-08-17T13:16:52.8093587Z OK (skipped=1) 2022-08-17T13:16:52.8093778Z 2022-08-17T13:16:52.8093932Z Generating XML reports... 2022-08-17T13:16:52.8125560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131652.xml 2022-08-17T13:16:53.9536411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:53.9537008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:53.9539272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:53.9539876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:54.1067437Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:54.1077465Z 2022-08-17T13:16:54.1077906Z Running tests... 2022-08-17T13:16:54.1078790Z ---------------------------------------------------------------------- 2022-08-17T13:16:54.1090661Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:54.1091142Z 2022-08-17T13:16:54.1091551Z ---------------------------------------------------------------------- 2022-08-17T13:16:54.1091989Z Ran 1 test in 0.001s 2022-08-17T13:16:54.1092202Z 2022-08-17T13:16:54.1092336Z OK (skipped=1) 2022-08-17T13:16:54.1092524Z 2022-08-17T13:16:54.1092685Z Generating XML reports... 2022-08-17T13:16:54.1121936Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131654.xml 2022-08-17T13:16:55.2622257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:55.2622946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:55.2625721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:55.2626311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:55.4148351Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:55.4158483Z 2022-08-17T13:16:55.4158796Z Running tests... 2022-08-17T13:16:55.4159412Z ---------------------------------------------------------------------- 2022-08-17T13:16:55.4166774Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:16:55.4167197Z 2022-08-17T13:16:55.4167616Z ---------------------------------------------------------------------- 2022-08-17T13:16:55.4168044Z Ran 1 test in 0.001s 2022-08-17T13:16:55.4168237Z 2022-08-17T13:16:55.4168360Z OK (skipped=1) 2022-08-17T13:16:55.4168523Z 2022-08-17T13:16:55.4168665Z Generating XML reports... 2022-08-17T13:16:55.4200184Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131655.xml 2022-08-17T13:16:56.5873506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:56.5873892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:56.5876938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:56.5877352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:56.7414362Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:56.7424237Z 2022-08-17T13:16:56.7424499Z Running tests... 2022-08-17T13:16:56.7424863Z ---------------------------------------------------------------------- 2022-08-17T13:16:56.7429683Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:56.7430102Z 2022-08-17T13:16:56.7430501Z ---------------------------------------------------------------------- 2022-08-17T13:16:56.7430955Z Ran 1 test in 0.001s 2022-08-17T13:16:56.7431150Z 2022-08-17T13:16:56.7431281Z OK (skipped=1) 2022-08-17T13:16:56.7431481Z 2022-08-17T13:16:56.7431641Z Generating XML reports... 2022-08-17T13:16:56.7463747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131656.xml 2022-08-17T13:16:57.9186717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:57.9187109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:57.9189890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:57.9190306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:58.0776083Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:58.0785380Z 2022-08-17T13:16:58.0785520Z Running tests... 2022-08-17T13:16:58.0785901Z ---------------------------------------------------------------------- 2022-08-17T13:16:58.0790947Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:16:58.0791356Z 2022-08-17T13:16:58.0791797Z ---------------------------------------------------------------------- 2022-08-17T13:16:58.0792258Z Ran 1 test in 0.001s 2022-08-17T13:16:58.0792399Z 2022-08-17T13:16:58.0792476Z OK (skipped=1) 2022-08-17T13:16:58.0792587Z 2022-08-17T13:16:58.0792677Z Generating XML reports... 2022-08-17T13:16:58.0822378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131658.xml 2022-08-17T13:16:59.2570155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:16:59.2570634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:16:59.2572238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:16:59.2572749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:16:59.4126331Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:16:59.4135375Z 2022-08-17T13:16:59.4135479Z Running tests... 2022-08-17T13:16:59.4135887Z ---------------------------------------------------------------------- 2022-08-17T13:16:59.6785198Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5456 2022-08-17T13:16:59.6812204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5457 2022-08-17T13:17:00.6583647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:00.6584064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:00.6586327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:00.6586856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:00.6957425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:00.6957779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:00.6960768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:00.6961144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:00.8121867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:00.8486731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:01.1848593Z skip: Need at least 2 CUDA devices (1.771s) 2022-08-17T13:17:01.1849157Z 2022-08-17T13:17:01.1850018Z ---------------------------------------------------------------------- 2022-08-17T13:17:01.1850564Z Ran 1 test in 1.771s 2022-08-17T13:17:01.1850702Z 2022-08-17T13:17:01.1850784Z OK (skipped=1) 2022-08-17T13:17:01.1850906Z 2022-08-17T13:17:01.1850996Z Generating XML reports... 2022-08-17T13:17:01.1887992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131659.xml 2022-08-17T13:17:02.5169806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:02.5170210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:02.5173407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:02.5173834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:02.6757192Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:02.6767230Z 2022-08-17T13:17:02.6767336Z Running tests... 2022-08-17T13:17:02.6767789Z ---------------------------------------------------------------------- 2022-08-17T13:17:02.9426025Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5489 2022-08-17T13:17:02.9454126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5490 2022-08-17T13:17:03.9103633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:03.9104040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:03.9106793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:03.9107347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:03.9107822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:03.9108191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:03.9111298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:03.9111661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:04.0665226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:04.0670508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:04.3489706Z skip: Need at least 2 CUDA devices (1.672s) 2022-08-17T13:17:04.3490068Z 2022-08-17T13:17:04.3490482Z ---------------------------------------------------------------------- 2022-08-17T13:17:04.3490890Z Ran 1 test in 1.672s 2022-08-17T13:17:04.3491110Z 2022-08-17T13:17:04.3491248Z OK (skipped=1) 2022-08-17T13:17:04.3491459Z 2022-08-17T13:17:04.3491589Z Generating XML reports... 2022-08-17T13:17:04.3526738Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131702.xml 2022-08-17T13:17:05.6673559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:05.6674228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:05.6676276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:05.6676660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:05.8231471Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:05.8241459Z 2022-08-17T13:17:05.8241562Z Running tests... 2022-08-17T13:17:05.8242011Z ---------------------------------------------------------------------- 2022-08-17T13:17:05.8248339Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:05.8248768Z 2022-08-17T13:17:05.8249041Z ---------------------------------------------------------------------- 2022-08-17T13:17:05.8249358Z Ran 1 test in 0.001s 2022-08-17T13:17:05.8249478Z 2022-08-17T13:17:05.8249551Z OK (skipped=1) 2022-08-17T13:17:05.8249664Z 2022-08-17T13:17:05.8249753Z Generating XML reports... 2022-08-17T13:17:05.8279412Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131705.xml 2022-08-17T13:17:06.9918801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:06.9919190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:06.9920932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:06.9921309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:07.1463735Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:07.1473605Z 2022-08-17T13:17:07.1473705Z Running tests... 2022-08-17T13:17:07.1474875Z ---------------------------------------------------------------------- 2022-08-17T13:17:07.1483877Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:07.1484389Z 2022-08-17T13:17:07.1484821Z ---------------------------------------------------------------------- 2022-08-17T13:17:07.1485100Z Ran 1 test in 0.001s 2022-08-17T13:17:07.1485209Z 2022-08-17T13:17:07.1485300Z OK (skipped=1) 2022-08-17T13:17:07.1485411Z 2022-08-17T13:17:07.1485501Z Generating XML reports... 2022-08-17T13:17:07.1517803Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131707.xml 2022-08-17T13:17:08.3394046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:08.3394421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:08.3396399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:08.3396792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:08.4945706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:08.4955323Z 2022-08-17T13:17:08.4955577Z Running tests... 2022-08-17T13:17:08.4956212Z ---------------------------------------------------------------------- 2022-08-17T13:17:08.4962352Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:08.4962623Z 2022-08-17T13:17:08.4962923Z ---------------------------------------------------------------------- 2022-08-17T13:17:08.4963207Z Ran 1 test in 0.001s 2022-08-17T13:17:08.4963327Z 2022-08-17T13:17:08.4963410Z OK (skipped=1) 2022-08-17T13:17:08.4963525Z 2022-08-17T13:17:08.4963670Z Generating XML reports... 2022-08-17T13:17:08.4994843Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131708.xml 2022-08-17T13:17:09.6772020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:09.6772394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:09.6774682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:09.6775262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:09.8330444Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:09.8339837Z 2022-08-17T13:17:09.8339974Z Running tests... 2022-08-17T13:17:09.8340713Z ---------------------------------------------------------------------- 2022-08-17T13:17:09.8346645Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:09.8347135Z 2022-08-17T13:17:09.8347558Z ---------------------------------------------------------------------- 2022-08-17T13:17:09.8347964Z Ran 1 test in 0.001s 2022-08-17T13:17:09.8348162Z 2022-08-17T13:17:09.8348285Z OK (skipped=1) 2022-08-17T13:17:09.8348483Z 2022-08-17T13:17:09.8348642Z Generating XML reports... 2022-08-17T13:17:09.8379837Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131709.xml 2022-08-17T13:17:11.0064386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:11.0065031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:11.0067287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:11.0067878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:11.1603684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:11.1614259Z 2022-08-17T13:17:11.1614677Z Running tests... 2022-08-17T13:17:11.1615092Z ---------------------------------------------------------------------- 2022-08-17T13:17:11.1620972Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:11.1621432Z 2022-08-17T13:17:11.1621845Z ---------------------------------------------------------------------- 2022-08-17T13:17:11.1622259Z Ran 1 test in 0.001s 2022-08-17T13:17:11.1622443Z 2022-08-17T13:17:11.1622568Z OK (skipped=1) 2022-08-17T13:17:11.1622766Z 2022-08-17T13:17:11.1622922Z Generating XML reports... 2022-08-17T13:17:11.1652776Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131711.xml 2022-08-17T13:17:12.3388324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:12.3388736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:12.3391196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:12.3391693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:12.4945599Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:12.4954535Z 2022-08-17T13:17:12.4954672Z Running tests... 2022-08-17T13:17:12.4955304Z ---------------------------------------------------------------------- 2022-08-17T13:17:12.4961156Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:12.4961770Z 2022-08-17T13:17:12.4962293Z ---------------------------------------------------------------------- 2022-08-17T13:17:12.4962630Z Ran 1 test in 0.001s 2022-08-17T13:17:12.4962754Z 2022-08-17T13:17:12.4963071Z OK (skipped=1) 2022-08-17T13:17:12.4963194Z 2022-08-17T13:17:12.4963271Z Generating XML reports... 2022-08-17T13:17:12.4995185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131712.xml 2022-08-17T13:17:13.6761289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:13.6761678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:13.6763914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:13.6764295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:13.8325661Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:13.8336229Z 2022-08-17T13:17:13.8336322Z Running tests... 2022-08-17T13:17:13.8336826Z ---------------------------------------------------------------------- 2022-08-17T13:17:13.8343195Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:13.8343646Z 2022-08-17T13:17:13.8344047Z ---------------------------------------------------------------------- 2022-08-17T13:17:13.8344554Z Ran 1 test in 0.001s 2022-08-17T13:17:13.8344702Z 2022-08-17T13:17:13.8344780Z OK (skipped=1) 2022-08-17T13:17:13.8344894Z 2022-08-17T13:17:13.8344984Z Generating XML reports... 2022-08-17T13:17:13.8375254Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131713.xml 2022-08-17T13:17:15.0250016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:15.0250563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:15.0251504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:15.0251965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:15.1805211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:15.1815584Z 2022-08-17T13:17:15.1815719Z Running tests... 2022-08-17T13:17:15.1816153Z ---------------------------------------------------------------------- 2022-08-17T13:17:15.1831066Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:15.1831443Z 2022-08-17T13:17:15.1831864Z ---------------------------------------------------------------------- 2022-08-17T13:17:15.1832275Z Ran 1 test in 0.001s 2022-08-17T13:17:15.1832396Z 2022-08-17T13:17:15.1832462Z OK (skipped=1) 2022-08-17T13:17:15.1832576Z 2022-08-17T13:17:15.1832665Z Generating XML reports... 2022-08-17T13:17:15.1861884Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131715.xml 2022-08-17T13:17:16.3670602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:16.3670986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:16.3673508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:16.3673888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:16.5237157Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:16.5246764Z 2022-08-17T13:17:16.5246876Z Running tests... 2022-08-17T13:17:16.5247385Z ---------------------------------------------------------------------- 2022-08-17T13:17:16.5260613Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:16.5261026Z 2022-08-17T13:17:16.5261824Z ---------------------------------------------------------------------- 2022-08-17T13:17:16.5262164Z Ran 1 test in 0.001s 2022-08-17T13:17:16.5262288Z 2022-08-17T13:17:16.5262368Z OK (skipped=1) 2022-08-17T13:17:16.5262484Z 2022-08-17T13:17:16.5262579Z Generating XML reports... 2022-08-17T13:17:16.5292095Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131716.xml 2022-08-17T13:17:17.7062264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:17.7062898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:17.7065746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:17.7066367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:17.8627070Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:17.8637600Z 2022-08-17T13:17:17.8637907Z Running tests... 2022-08-17T13:17:17.8638518Z ---------------------------------------------------------------------- 2022-08-17T13:17:17.8646018Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:17.8646418Z 2022-08-17T13:17:17.8646730Z ---------------------------------------------------------------------- 2022-08-17T13:17:17.8646993Z Ran 1 test in 0.001s 2022-08-17T13:17:17.8647117Z 2022-08-17T13:17:17.8647195Z OK (skipped=1) 2022-08-17T13:17:17.8647335Z 2022-08-17T13:17:17.8647455Z Generating XML reports... 2022-08-17T13:17:17.8677881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220817131717.xml 2022-08-17T13:17:19.0511162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:19.0511784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:19.0514067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:19.0514689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:19.2069299Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:19.2079654Z 2022-08-17T13:17:19.2080093Z Running tests... 2022-08-17T13:17:19.2080511Z ---------------------------------------------------------------------- 2022-08-17T13:17:19.2093945Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:19.2094443Z 2022-08-17T13:17:19.2094868Z ---------------------------------------------------------------------- 2022-08-17T13:17:19.2095121Z Ran 1 test in 0.001s 2022-08-17T13:17:19.2095245Z 2022-08-17T13:17:19.2095322Z OK (skipped=1) 2022-08-17T13:17:19.2095450Z 2022-08-17T13:17:19.2095539Z Generating XML reports... 2022-08-17T13:17:19.2124924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131719.xml 2022-08-17T13:17:20.3945802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:20.3946183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:20.3948417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:20.3948794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:20.5504054Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:20.5513146Z 2022-08-17T13:17:20.5513294Z Running tests... 2022-08-17T13:17:20.5513891Z ---------------------------------------------------------------------- 2022-08-17T13:17:20.5524739Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:20.5525199Z 2022-08-17T13:17:20.5525661Z ---------------------------------------------------------------------- 2022-08-17T13:17:20.5526073Z Ran 1 test in 0.001s 2022-08-17T13:17:20.5526195Z 2022-08-17T13:17:20.5526284Z OK (skipped=1) 2022-08-17T13:17:20.5526400Z 2022-08-17T13:17:20.5526492Z Generating XML reports... 2022-08-17T13:17:20.5558888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131720.xml 2022-08-17T13:17:21.7341359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:21.7341757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:21.7344072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:21.7344497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:21.8889305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:21.8899034Z 2022-08-17T13:17:21.8899162Z Running tests... 2022-08-17T13:17:21.8900079Z ---------------------------------------------------------------------- 2022-08-17T13:17:21.8913841Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:21.8914231Z 2022-08-17T13:17:21.8914582Z ---------------------------------------------------------------------- 2022-08-17T13:17:21.8915039Z Ran 1 test in 0.001s 2022-08-17T13:17:21.8915250Z 2022-08-17T13:17:21.8915392Z OK (skipped=1) 2022-08-17T13:17:21.8915602Z 2022-08-17T13:17:21.8915766Z Generating XML reports... 2022-08-17T13:17:21.8944872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131721.xml 2022-08-17T13:17:23.0589516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:23.0590062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:23.0591058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:23.0591431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:23.2142393Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:23.2151916Z 2022-08-17T13:17:23.2152019Z Running tests... 2022-08-17T13:17:23.2152928Z ---------------------------------------------------------------------- 2022-08-17T13:17:23.2177197Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:17:23.2177633Z 2022-08-17T13:17:23.2178043Z ---------------------------------------------------------------------- 2022-08-17T13:17:23.2178488Z Ran 1 test in 0.003s 2022-08-17T13:17:23.2178686Z 2022-08-17T13:17:23.2178815Z OK (skipped=1) 2022-08-17T13:17:23.2179001Z 2022-08-17T13:17:23.2179139Z Generating XML reports... 2022-08-17T13:17:23.2210090Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131723.xml 2022-08-17T13:17:24.3899071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:24.3899770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:24.3900925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:24.3901560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:24.5428952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:24.5439527Z 2022-08-17T13:17:24.5440049Z Running tests... 2022-08-17T13:17:24.5440626Z ---------------------------------------------------------------------- 2022-08-17T13:17:24.5455149Z test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:24.5455564Z 2022-08-17T13:17:24.5455976Z ---------------------------------------------------------------------- 2022-08-17T13:17:24.5456400Z Ran 1 test in 0.002s 2022-08-17T13:17:24.5456602Z 2022-08-17T13:17:24.5456715Z OK (skipped=1) 2022-08-17T13:17:24.5456913Z 2022-08-17T13:17:24.5457076Z Generating XML reports... 2022-08-17T13:17:24.5486779Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131724.xml 2022-08-17T13:17:25.7079683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:25.7080320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:25.7082774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:25.7083378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:25.8630666Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:25.8641526Z 2022-08-17T13:17:25.8641862Z Running tests... 2022-08-17T13:17:25.8642447Z ---------------------------------------------------------------------- 2022-08-17T13:17:25.8658876Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:17:25.8659304Z 2022-08-17T13:17:25.8659750Z ---------------------------------------------------------------------- 2022-08-17T13:17:25.8660192Z Ran 1 test in 0.002s 2022-08-17T13:17:25.8660409Z 2022-08-17T13:17:25.8660547Z OK (skipped=1) 2022-08-17T13:17:25.8660750Z 2022-08-17T13:17:25.8660910Z Generating XML reports... 2022-08-17T13:17:25.8690088Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131725.xml 2022-08-17T13:17:27.0404463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:27.0404868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:27.0407463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:27.0407951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:27.1941232Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:27.1951117Z 2022-08-17T13:17:27.1951271Z Running tests... 2022-08-17T13:17:27.1951903Z ---------------------------------------------------------------------- 2022-08-17T13:17:27.1967179Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:27.1967639Z 2022-08-17T13:17:27.1968114Z ---------------------------------------------------------------------- 2022-08-17T13:17:27.1968439Z Ran 1 test in 0.002s 2022-08-17T13:17:27.1968563Z 2022-08-17T13:17:27.1968640Z OK (skipped=1) 2022-08-17T13:17:27.1968743Z 2022-08-17T13:17:27.1968841Z Generating XML reports... 2022-08-17T13:17:27.1999086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131727.xml 2022-08-17T13:17:28.3617613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:28.3618009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:28.3620641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:28.3621013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:28.5175926Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:28.5186328Z 2022-08-17T13:17:28.5186490Z Running tests... 2022-08-17T13:17:28.5187084Z ---------------------------------------------------------------------- 2022-08-17T13:17:28.5206595Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:17:28.5206956Z 2022-08-17T13:17:28.5207797Z ---------------------------------------------------------------------- 2022-08-17T13:17:28.5208236Z Ran 1 test in 0.002s 2022-08-17T13:17:28.5208364Z 2022-08-17T13:17:28.5208446Z OK (skipped=1) 2022-08-17T13:17:28.5208566Z 2022-08-17T13:17:28.5208655Z Generating XML reports... 2022-08-17T13:17:28.5241606Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131728.xml 2022-08-17T13:17:29.7228454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:29.7228854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:29.7231018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:29.7231676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:29.8810370Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:29.8820267Z 2022-08-17T13:17:29.8820415Z Running tests... 2022-08-17T13:17:29.8820849Z ---------------------------------------------------------------------- 2022-08-17T13:17:29.8836454Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:29.8836721Z 2022-08-17T13:17:29.8837216Z ---------------------------------------------------------------------- 2022-08-17T13:17:29.8837681Z Ran 1 test in 0.002s 2022-08-17T13:17:29.8837940Z 2022-08-17T13:17:29.8838018Z OK (skipped=1) 2022-08-17T13:17:29.8838129Z 2022-08-17T13:17:29.8838218Z Generating XML reports... 2022-08-17T13:17:29.8867574Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131729.xml 2022-08-17T13:17:31.0611968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:31.0612343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:31.0614637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:31.0615128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:31.2160333Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:31.2170892Z 2022-08-17T13:17:31.2171012Z Running tests... 2022-08-17T13:17:31.2171469Z ---------------------------------------------------------------------- 2022-08-17T13:17:31.2189812Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:17:31.2190050Z 2022-08-17T13:17:31.2190281Z ---------------------------------------------------------------------- 2022-08-17T13:17:31.2190624Z Ran 1 test in 0.002s 2022-08-17T13:17:31.2190765Z 2022-08-17T13:17:31.2190841Z OK (skipped=1) 2022-08-17T13:17:31.2190955Z 2022-08-17T13:17:31.2191042Z Generating XML reports... 2022-08-17T13:17:31.2223218Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131731.xml 2022-08-17T13:17:32.3929256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:32.3929664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:32.3931581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:32.3932311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:32.5471852Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:32.5481720Z 2022-08-17T13:17:32.5481812Z Running tests... 2022-08-17T13:17:32.5482298Z ---------------------------------------------------------------------- 2022-08-17T13:17:32.5499651Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:32.5500113Z 2022-08-17T13:17:32.5500570Z ---------------------------------------------------------------------- 2022-08-17T13:17:32.5501036Z Ran 1 test in 0.002s 2022-08-17T13:17:32.5501248Z 2022-08-17T13:17:32.5501365Z OK (skipped=1) 2022-08-17T13:17:32.5501558Z 2022-08-17T13:17:32.5501691Z Generating XML reports... 2022-08-17T13:17:32.5530484Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131732.xml 2022-08-17T13:17:33.7234914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:33.7235563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:33.7238640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:33.7239273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:33.8783757Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:33.8794093Z 2022-08-17T13:17:33.8794382Z Running tests... 2022-08-17T13:17:33.8794993Z ---------------------------------------------------------------------- 2022-08-17T13:17:33.8808505Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:33.8808981Z 2022-08-17T13:17:33.8809314Z ---------------------------------------------------------------------- 2022-08-17T13:17:33.8809609Z Ran 1 test in 0.001s 2022-08-17T13:17:33.8809732Z 2022-08-17T13:17:33.8809809Z OK (skipped=1) 2022-08-17T13:17:33.8809985Z 2022-08-17T13:17:33.8810086Z Generating XML reports... 2022-08-17T13:17:33.8840248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131733.xml 2022-08-17T13:17:35.0508872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:35.0509276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:35.0511599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:35.0512024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:35.2062375Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:35.2072042Z 2022-08-17T13:17:35.2072157Z Running tests... 2022-08-17T13:17:35.2072775Z ---------------------------------------------------------------------- 2022-08-17T13:17:35.2083258Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:35.2083609Z 2022-08-17T13:17:35.2083997Z ---------------------------------------------------------------------- 2022-08-17T13:17:35.2084264Z Ran 1 test in 0.001s 2022-08-17T13:17:35.2084387Z 2022-08-17T13:17:35.2084466Z OK (skipped=1) 2022-08-17T13:17:35.2084570Z 2022-08-17T13:17:35.2084662Z Generating XML reports... 2022-08-17T13:17:35.2115504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131735.xml 2022-08-17T13:17:36.4010391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:36.4011023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:36.4013587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:36.4014137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:36.5570266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:36.5579911Z 2022-08-17T13:17:36.5580047Z Running tests... 2022-08-17T13:17:36.5580664Z ---------------------------------------------------------------------- 2022-08-17T13:17:36.5614286Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-08-17T13:17:36.5614742Z 2022-08-17T13:17:36.5615052Z ---------------------------------------------------------------------- 2022-08-17T13:17:36.5615321Z Ran 1 test in 0.003s 2022-08-17T13:17:36.5615444Z 2022-08-17T13:17:36.5615523Z OK (skipped=1) 2022-08-17T13:17:36.5615647Z 2022-08-17T13:17:36.5615723Z Generating XML reports... 2022-08-17T13:17:36.5644695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131736.xml 2022-08-17T13:17:37.7251946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:37.7252581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:37.7254276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:37.7254918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:37.8771007Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:37.8781301Z 2022-08-17T13:17:37.8781783Z Running tests... 2022-08-17T13:17:37.8782199Z ---------------------------------------------------------------------- 2022-08-17T13:17:37.8797710Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:37.8798132Z 2022-08-17T13:17:37.8798569Z ---------------------------------------------------------------------- 2022-08-17T13:17:37.8798864Z Ran 1 test in 0.002s 2022-08-17T13:17:37.8798984Z 2022-08-17T13:17:37.8799057Z OK (skipped=1) 2022-08-17T13:17:37.8799171Z 2022-08-17T13:17:37.8799260Z Generating XML reports... 2022-08-17T13:17:37.8827579Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131737.xml 2022-08-17T13:17:39.0381239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:39.0381897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:39.0382716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:39.0383445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:39.1928245Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:39.1939033Z 2022-08-17T13:17:39.1939375Z Running tests... 2022-08-17T13:17:39.1940000Z ---------------------------------------------------------------------- 2022-08-17T13:17:39.1955021Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:39.1955480Z 2022-08-17T13:17:39.1955955Z ---------------------------------------------------------------------- 2022-08-17T13:17:39.1956241Z Ran 1 test in 0.002s 2022-08-17T13:17:39.1956348Z 2022-08-17T13:17:39.1956426Z OK (skipped=1) 2022-08-17T13:17:39.1956541Z 2022-08-17T13:17:39.1956631Z Generating XML reports... 2022-08-17T13:17:39.1987633Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131739.xml 2022-08-17T13:17:40.3808617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:40.3809072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:40.3810513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:40.3810987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:40.5357547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:40.5367721Z 2022-08-17T13:17:40.5367829Z Running tests... 2022-08-17T13:17:40.5368549Z ---------------------------------------------------------------------- 2022-08-17T13:17:40.5387521Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-08-17T13:17:40.5387941Z 2022-08-17T13:17:40.5388285Z ---------------------------------------------------------------------- 2022-08-17T13:17:40.5388763Z Ran 1 test in 0.002s 2022-08-17T13:17:40.5388982Z 2022-08-17T13:17:40.5389090Z OK (skipped=1) 2022-08-17T13:17:40.5389210Z 2022-08-17T13:17:40.5389286Z Generating XML reports... 2022-08-17T13:17:40.5419989Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131740.xml 2022-08-17T13:17:41.7240060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:41.7240457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:41.7242661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:41.7243178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:41.8795782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:41.8805788Z 2022-08-17T13:17:41.8806199Z Running tests... 2022-08-17T13:17:41.8806570Z ---------------------------------------------------------------------- 2022-08-17T13:17:41.8817767Z test_send_recv (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:17:41.8818177Z 2022-08-17T13:17:41.8818711Z ---------------------------------------------------------------------- 2022-08-17T13:17:41.8819010Z Ran 1 test in 0.001s 2022-08-17T13:17:41.8819143Z 2022-08-17T13:17:41.8819226Z OK (skipped=1) 2022-08-17T13:17:41.8819388Z 2022-08-17T13:17:41.8819484Z Generating XML reports... 2022-08-17T13:17:41.8850187Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131741.xml 2022-08-17T13:17:43.0607502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:43.0607896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:43.0609942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:43.0610407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:43.2163129Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:43.2174215Z 2022-08-17T13:17:43.2174529Z Running tests... 2022-08-17T13:17:43.2174903Z ---------------------------------------------------------------------- 2022-08-17T13:17:43.2211338Z test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-08-17T13:17:43.2211717Z 2022-08-17T13:17:43.2212068Z ---------------------------------------------------------------------- 2022-08-17T13:17:43.2212400Z Ran 1 test in 0.004s 2022-08-17T13:17:43.2212525Z 2022-08-17T13:17:43.2212588Z OK (skipped=1) 2022-08-17T13:17:43.2212702Z 2022-08-17T13:17:43.2212797Z Generating XML reports... 2022-08-17T13:17:43.2244643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220817131743.xml 2022-08-17T13:17:44.3938354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:44.3938993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:44.3941200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:44.3941783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:44.5460479Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-08-17T13:17:44.5470963Z 2022-08-17T13:17:44.5471489Z Running tests... 2022-08-17T13:17:44.5471956Z ---------------------------------------------------------------------- 2022-08-17T13:17:44.5477075Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:17:44.5477480Z 2022-08-17T13:17:44.5477883Z ---------------------------------------------------------------------- 2022-08-17T13:17:44.5478289Z Ran 1 test in 0.001s 2022-08-17T13:17:44.5478411Z 2022-08-17T13:17:44.5478486Z OK (skipped=1) 2022-08-17T13:17:44.5478598Z 2022-08-17T13:17:44.5478686Z Generating XML reports... 2022-08-17T13:17:44.5508752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220817131744.xml 2022-08-17T13:17:44.9868054Z Running distributed/test_c10d_object_collectives ... [2022-08-17 13:17:44.986324] 2022-08-17T13:17:44.9868692Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:17:44.986412] 2022-08-17T13:17:46.0759571Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives 2022-08-17T13:17:46.0770893Z 2022-08-17T13:17:46.0771041Z Running tests... 2022-08-17T13:17:46.0771670Z ---------------------------------------------------------------------- 2022-08-17T13:17:46.3533661Z test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5822 2022-08-17T13:17:46.3561009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5823 2022-08-17T13:17:47.3148468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:47.3149140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:47.3149954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:47.3150610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:47.3270211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:47.3270842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:47.3272782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:47.3273410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:47.4697045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:47.4817898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:47.5007497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:17:47.5008118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:17:47.5009120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:47.5009982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:47.8594665Z ok (1.782s) 2022-08-17T13:17:47.8649574Z test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5850 2022-08-17T13:17:47.8688469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5851 2022-08-17T13:17:48.8311348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:48.8311822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:48.8313986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:48.8314351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:48.8517741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:48.8518116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:48.8520080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:48.8520680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:48.9871094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:49.0069027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:49.0181865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:17:49.0182376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:17:49.0183099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:49.0183678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:49.3719716Z ok (1.512s) 2022-08-17T13:17:49.3778535Z test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5878 2022-08-17T13:17:49.3816841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5879 2022-08-17T13:17:50.3387396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:50.3388063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:50.3389655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:50.3390274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:50.3441257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:50.3441924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:50.3443948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:50.3444632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:50.4932288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:50.4985234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:50.5094819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:17:50.5095256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:17:50.5095921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:50.5096682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:50.7851273Z ok (1.413s) 2022-08-17T13:17:50.7906346Z test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5906 2022-08-17T13:17:50.7946894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5907 2022-08-17T13:17:51.7683746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:51.7684260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:51.7685583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:51.7686122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:51.7927873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:51.7928225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:51.7931431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:51.7931810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:51.9210274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:51.9462475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:17:51.9624372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:17:51.9624783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:17:51.9625432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:51.9626014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-08-17T13:17:52.2980888Z ok (1.513s) 2022-08-17T13:17:52.2981376Z 2022-08-17T13:17:52.2982400Z ---------------------------------------------------------------------- 2022-08-17T13:17:52.2982850Z Ran 4 tests in 6.221s 2022-08-17T13:17:52.2983130Z 2022-08-17T13:17:52.2983213Z OK 2022-08-17T13:17:52.2983374Z 2022-08-17T13:17:52.2983530Z Generating XML reports... 2022-08-17T13:17:52.3023096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220817131746.xml 2022-08-17T13:17:52.6328280Z Running distributed/test_c10d_pypg ... [2022-08-17 13:17:52.632427] 2022-08-17T13:17:53.7260154Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:17:52.632509] 2022-08-17T13:17:53.7260734Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg 2022-08-17T13:17:53.7274460Z 2022-08-17T13:17:53.7274562Z Running tests... 2022-08-17T13:17:53.7275754Z ---------------------------------------------------------------------- 2022-08-17T13:17:53.7284082Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:17:53.9897250Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5945 2022-08-17T13:17:54.9327416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:54.9327812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:54.9330361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:54.9330991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:55.0867857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:55.3929276Z skip: Need at least 2 CUDA devices (1.665s) 2022-08-17T13:17:55.3939921Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:17:55.4010450Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5956 2022-08-17T13:17:56.3491114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:56.3491505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:56.3494306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:56.3494823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:56.5032893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:56.8039222Z skip: Need at least 2 CUDA devices (1.411s) 2022-08-17T13:17:56.8048355Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:17:56.8098327Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5967 2022-08-17T13:17:57.7335646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:57.7336056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:57.7338477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:57.7338843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:57.8831780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:58.1125218Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:17:58.1134097Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:17:58.1180816Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5978 2022-08-17T13:17:59.0541654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:17:59.0542041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:17:59.0545304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:17:59.0545666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:17:59.2072517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:17:59.4208802Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:17:59.4216109Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:17:59.4263862Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5989 2022-08-17T13:18:00.3627082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:00.3627474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:00.3629885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:00.3630404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:00.5136039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:00.7292434Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:00.7299048Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:00.7345798Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6000 2022-08-17T13:18:01.6759375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:01.6760040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:01.6762242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:01.6762851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:01.8260580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:02.0376179Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:02.0386061Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:02.0433473Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6011 2022-08-17T13:18:02.9846037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:02.9846461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:02.9848210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:02.9848585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:03.1374602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:03.3464465Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:03.3474744Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:03.3522667Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6022 2022-08-17T13:18:04.2910024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:04.2910431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:04.2912582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:04.2913008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:04.4409072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:04.6553054Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:04.6558508Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:04.6601468Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6033 2022-08-17T13:18:05.5890933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:05.5891326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:05.5893725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:05.5894241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:05.7403862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:05.9632506Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:05.9642393Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:05.9690173Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6044 2022-08-17T13:18:06.9138312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:06.9138982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:06.9141319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:06.9141910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:07.0639123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:07.2720220Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:07.2729854Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:07.2778278Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6055 2022-08-17T13:18:08.2349513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:08.2350156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:08.2352152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:08.2352676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:08.3878918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:08.6811853Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:08.6822361Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:08.6870152Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6066 2022-08-17T13:18:09.6282209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:09.6282588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:09.6284719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:09.6285098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:09.7813529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:09.9900647Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:09.9912067Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-08-17T13:18:09.9960175Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6077 2022-08-17T13:18:10.9420259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:10.9420645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:10.9423458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:10.9423840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:11.0973294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:11.3989120Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:11.4046898Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6088 2022-08-17T13:18:12.3439040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:12.3439675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:12.3442248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:12.3442621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:12.4953154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:12.5026307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeau0i0ft 2022-08-17T13:18:12.5028026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeau0i0ft/_remote_module_non_scriptable.py 2022-08-17T13:18:12.8075705Z ok (1.409s) 2022-08-17T13:18:12.8128767Z test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6099 2022-08-17T13:18:13.7546006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:13.7546817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:13.7570674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:13.7571390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:13.9124238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:13.9198240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon2a21oz 2022-08-17T13:18:13.9200005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon2a21oz/_remote_module_non_scriptable.py 2022-08-17T13:18:13.9339941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:18:14.2158475Z ok (1.408s) 2022-08-17T13:18:14.2211578Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6110 2022-08-17T13:18:15.1668415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:15.1668852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:15.1670895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:15.1671409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:15.3211408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:15.3286081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bt7j3dw 2022-08-17T13:18:15.3287699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bt7j3dw/_remote_module_non_scriptable.py 2022-08-17T13:18:15.3434179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:18:15.6240505Z ok (1.408s) 2022-08-17T13:18:15.6297643Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6121 2022-08-17T13:18:16.5712418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:16.5712816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:16.5715253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:16.5715634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:16.7212555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:16.7217813Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.7219410Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.7220387Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.7221304Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.7222331Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.7223435Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:16.9325033Z ok (1.308s) 2022-08-17T13:18:16.9430462Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6132 2022-08-17T13:18:17.8810270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:17.8810976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:17.8813836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:17.8814440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:18.0416672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:18.3459218Z skip: Need at least 2 CUDA devices (1.413s) 2022-08-17T13:18:18.3519621Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6143 2022-08-17T13:18:19.2864338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:19.2864729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:19.2867657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:19.2868157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:19.4364030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:19.6546797Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:19.6554075Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:19.6601437Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6154 2022-08-17T13:18:20.5956435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:20.5956827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:20.5959441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:20.5960002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:20.7453088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:20.9627539Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:20.9634732Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:20.9682393Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6165 2022-08-17T13:18:21.9113811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:21.9114212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:21.9116735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:21.9117200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:22.0644465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:22.3710475Z skip: Need at least 2 CUDA devices (1.408s) 2022-08-17T13:18:22.3719147Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:22.3769770Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6176 2022-08-17T13:18:23.3153663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:23.3154187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:23.3156670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:23.3157094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:23.4655291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:23.6798419Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:23.6806483Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:23.6854293Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6187 2022-08-17T13:18:24.6248348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:24.6248761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:24.6251259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:24.6251787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:24.7772370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:24.9882433Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:24.9890352Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:24.9939284Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6198 2022-08-17T13:18:25.9467057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:25.9467743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:25.9470040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:25.9470528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:26.1025253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:26.3969138Z skip: Need at least 2 CUDA devices (1.408s) 2022-08-17T13:18:26.3975720Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:26.4025206Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6209 2022-08-17T13:18:27.3446251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:27.3446653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:27.3449094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:27.3449576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:27.4936163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:27.7052240Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:27.7063133Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:27.7111155Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6220 2022-08-17T13:18:28.6597189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:28.6597887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:28.6600613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:28.6601200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:28.8171517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:29.1139587Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:29.1149888Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:29.1198601Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6231 2022-08-17T13:18:30.0542370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:30.0542762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:30.0545399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:30.0545776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:30.2058557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:30.4226870Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:30.4233393Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:30.4281780Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6242 2022-08-17T13:18:31.3742184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:31.3742637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:31.3745242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:31.3745914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:31.5253738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:31.7309147Z skip: Need at least 2 CUDA devices (1.308s) 2022-08-17T13:18:31.7319344Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:31.7367907Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6253 2022-08-17T13:18:32.6824271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:32.6825031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:32.6828090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:32.6828724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:32.8350039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:33.1397589Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:33.1407574Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:33.1455697Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6264 2022-08-17T13:18:34.0916142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:34.0916512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:34.0919494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:34.0919870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:34.2434087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:34.5483909Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:34.5495118Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:34.5543603Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6275 2022-08-17T13:18:35.5008837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:35.5009493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:35.5015874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:35.5016294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:35.6535563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:35.9573219Z skip: Need at least 2 CUDA devices (1.409s) 2022-08-17T13:18:35.9585000Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-08-17T13:18:35.9647815Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6286 2022-08-17T13:18:36.9179719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:36.9180117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:36.9182756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:36.9183189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:37.0726669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:37.3676427Z skip: Need at least 2 CUDA devices (1.410s) 2022-08-17T13:18:37.3734873Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6297 2022-08-17T13:18:38.3193604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:38.3194002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:38.3196368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:38.3196742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:38.4754682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:38.4830986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9fl7iqv 2022-08-17T13:18:38.4832669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9fl7iqv/_remote_module_non_scriptable.py 2022-08-17T13:18:38.7764586Z ok (1.409s) 2022-08-17T13:18:38.7818828Z test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6308 2022-08-17T13:18:39.7363866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:39.7364278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:39.7366983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:39.7367500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:39.8912085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:39.8986522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_l7o0gb 2022-08-17T13:18:39.8988177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_l7o0gb/_remote_module_non_scriptable.py 2022-08-17T13:18:39.9129729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:18:40.1848444Z ok (1.408s) 2022-08-17T13:18:40.1901550Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6319 2022-08-17T13:18:41.1363260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:41.1363696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:41.1366473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:41.1366902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:41.2895178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:41.2970654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnzgwcsb 2022-08-17T13:18:41.2972192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnzgwcsb/_remote_module_non_scriptable.py 2022-08-17T13:18:41.3113002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:18:41.5930835Z ok (1.408s) 2022-08-17T13:18:41.5980992Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6330 2022-08-17T13:18:42.5378433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:42.5379132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:42.5381673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:42.5382593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:42.6955724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:42.6960741Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:42.6962078Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:42.6963414Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:42.6965294Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:42.6967189Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:42.6968814Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-08-17T13:18:43.0010360Z ok (1.408s) 2022-08-17T13:18:43.0070333Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6341 2022-08-17T13:18:43.9397385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:43.9401145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:43.9401901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:43.9402300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:44.0902206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:44.3096823Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:44.3156817Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6352 2022-08-17T13:18:45.2590400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:45.2590794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:45.2593399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:45.2593769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:45.4103268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:18:45.6184990Z skip: Need at least 2 CUDA devices (1.309s) 2022-08-17T13:18:45.6185288Z 2022-08-17T13:18:45.6186758Z ---------------------------------------------------------------------- 2022-08-17T13:18:45.6187105Z Ran 38 tests in 51.891s 2022-08-17T13:18:45.6187254Z 2022-08-17T13:18:45.6187322Z OK (skipped=30) 2022-08-17T13:18:45.6187436Z 2022-08-17T13:18:45.6187527Z Generating XML reports... 2022-08-17T13:18:45.6240178Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20220817131753.xml 2022-08-17T13:18:45.6260352Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20220817131753.xml 2022-08-17T13:18:45.9541137Z Running distributed/test_c10d_spawn_gloo ... [2022-08-17 13:18:45.953722] 2022-08-17T13:18:45.9541750Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:18:45.953810] 2022-08-17T13:18:47.0437158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0crlgfms 2022-08-17T13:18:47.0438014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0crlgfms/_remote_module_non_scriptable.py 2022-08-17T13:18:47.2944768Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-08-17T13:18:47.2945578Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-08-17T13:18:47.2946149Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-08-17T13:18:47.2946646Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-08-17T13:18:47.2947651Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-08-17T13:18:47.2948333Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:18:47.2948657Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:18:47.2948981Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:18:47.2949302Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:18:47.2949586Z 2022-08-17T13:18:47.2949821Z 2022-08-17T13:18:47.2950757Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-08-17T13:18:47.2951691Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2951997Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2952313Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2952616Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2952922Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2953225Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2953510Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:47.2953810Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-08-17T13:18:48.2310740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:48.2311136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:48.2312967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:48.2313346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:48.3809258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28nret1_ 2022-08-17T13:18:48.3809971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28nret1_/_remote_module_non_scriptable.py 2022-08-17T13:18:48.6290714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:48.6301906Z 2022-08-17T13:18:48.6302210Z Running tests... 2022-08-17T13:18:48.6302926Z ---------------------------------------------------------------------- 2022-08-17T13:18:48.6349930Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:18:48.6420468Z ok (0.012s) 2022-08-17T13:18:48.6421186Z 2022-08-17T13:18:48.6421806Z ---------------------------------------------------------------------- 2022-08-17T13:18:48.6422266Z Ran 1 test in 0.012s 2022-08-17T13:18:48.6422464Z 2022-08-17T13:18:48.6422552Z OK 2022-08-17T13:18:48.6422699Z 2022-08-17T13:18:48.6422957Z Generating XML reports... 2022-08-17T13:18:48.6447887Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131848.xml 2022-08-17T13:18:49.8765661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:49.8766521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:49.8767808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:49.8768427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:50.0286931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliwewuxi 2022-08-17T13:18:50.0287905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliwewuxi/_remote_module_non_scriptable.py 2022-08-17T13:18:50.2810527Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:50.2820930Z 2022-08-17T13:18:50.2821152Z Running tests... 2022-08-17T13:18:50.2821770Z ---------------------------------------------------------------------- 2022-08-17T13:18:50.2826549Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s) 2022-08-17T13:18:50.2827106Z 2022-08-17T13:18:50.2827620Z ---------------------------------------------------------------------- 2022-08-17T13:18:50.2828087Z Ran 1 test in 0.001s 2022-08-17T13:18:50.2828304Z 2022-08-17T13:18:50.2828405Z OK (skipped=1) 2022-08-17T13:18:50.2828520Z 2022-08-17T13:18:50.2828610Z Generating XML reports... 2022-08-17T13:18:50.2853262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131850.xml 2022-08-17T13:18:51.5245319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:51.5246216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:51.5248005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:51.5248568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:51.6752280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsawxd0ga 2022-08-17T13:18:51.6753243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsawxd0ga/_remote_module_non_scriptable.py 2022-08-17T13:18:51.9248522Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:51.9259227Z 2022-08-17T13:18:51.9259515Z Running tests... 2022-08-17T13:18:51.9260173Z ---------------------------------------------------------------------- 2022-08-17T13:18:51.9274012Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s) 2022-08-17T13:18:51.9274322Z 2022-08-17T13:18:51.9275020Z ---------------------------------------------------------------------- 2022-08-17T13:18:51.9275449Z Ran 1 test in 0.001s 2022-08-17T13:18:51.9275646Z 2022-08-17T13:18:51.9275782Z OK (skipped=1) 2022-08-17T13:18:51.9275969Z 2022-08-17T13:18:51.9276119Z Generating XML reports... 2022-08-17T13:18:51.9301363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131851.xml 2022-08-17T13:18:53.1584554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:53.1585259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:53.1586793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:53.1587391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:53.3102979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6yce2ab 2022-08-17T13:18:53.3103687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6yce2ab/_remote_module_non_scriptable.py 2022-08-17T13:18:53.5634643Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:53.5646023Z 2022-08-17T13:18:53.5646638Z Running tests... 2022-08-17T13:18:53.5647099Z ---------------------------------------------------------------------- 2022-08-17T13:18:53.5652203Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:18:53.5652625Z 2022-08-17T13:18:53.5652907Z ---------------------------------------------------------------------- 2022-08-17T13:18:53.5653153Z Ran 1 test in 0.001s 2022-08-17T13:18:53.5653275Z 2022-08-17T13:18:53.5653352Z OK (skipped=1) 2022-08-17T13:18:53.5653467Z 2022-08-17T13:18:53.5653560Z Generating XML reports... 2022-08-17T13:18:53.5678027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131853.xml 2022-08-17T13:18:54.7919415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:54.7920061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:54.7921519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:54.7922164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:54.9443303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsj43q_sl 2022-08-17T13:18:54.9444052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsj43q_sl/_remote_module_non_scriptable.py 2022-08-17T13:18:55.1923932Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:55.1935346Z 2022-08-17T13:18:55.1935914Z Running tests... 2022-08-17T13:18:55.1936328Z ---------------------------------------------------------------------- 2022-08-17T13:18:55.1941488Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.001s) 2022-08-17T13:18:55.1942110Z 2022-08-17T13:18:55.1942547Z ---------------------------------------------------------------------- 2022-08-17T13:18:55.1943145Z Ran 1 test in 0.001s 2022-08-17T13:18:55.1943267Z 2022-08-17T13:18:55.1943349Z OK (skipped=1) 2022-08-17T13:18:55.1943473Z 2022-08-17T13:18:55.1943565Z Generating XML reports... 2022-08-17T13:18:55.1968653Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131855.xml 2022-08-17T13:18:56.4098408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:56.4099106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:56.4100125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:56.4100533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:56.5598123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu8ezb_q 2022-08-17T13:18:56.5599096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu8ezb_q/_remote_module_non_scriptable.py 2022-08-17T13:18:56.8101191Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:56.8111980Z 2022-08-17T13:18:56.8112075Z Running tests... 2022-08-17T13:18:56.8112768Z ---------------------------------------------------------------------- 2022-08-17T13:18:56.8118134Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:18:56.8118386Z 2022-08-17T13:18:56.8118616Z ---------------------------------------------------------------------- 2022-08-17T13:18:56.8118887Z Ran 1 test in 0.001s 2022-08-17T13:18:56.8119036Z 2022-08-17T13:18:56.8119114Z OK (skipped=1) 2022-08-17T13:18:56.8119231Z 2022-08-17T13:18:56.8119322Z Generating XML reports... 2022-08-17T13:18:56.8143396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131856.xml 2022-08-17T13:18:58.0393343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:58.0393730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:58.0395540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:58.0395918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:58.1923497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6dw3gpx 2022-08-17T13:18:58.1924707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6dw3gpx/_remote_module_non_scriptable.py 2022-08-17T13:18:58.4457530Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:18:58.4468202Z 2022-08-17T13:18:58.4468336Z Running tests... 2022-08-17T13:18:58.4468753Z ---------------------------------------------------------------------- 2022-08-17T13:18:58.4474200Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:18:58.4474679Z 2022-08-17T13:18:58.4475131Z ---------------------------------------------------------------------- 2022-08-17T13:18:58.4475602Z Ran 1 test in 0.001s 2022-08-17T13:18:58.4475811Z 2022-08-17T13:18:58.4475892Z OK (skipped=1) 2022-08-17T13:18:58.4475993Z 2022-08-17T13:18:58.4476092Z Generating XML reports... 2022-08-17T13:18:58.4500369Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131858.xml 2022-08-17T13:18:59.6964643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:18:59.6965261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:18:59.6966492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:18:59.6967132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:18:59.8474187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17qkjhep 2022-08-17T13:18:59.8474730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17qkjhep/_remote_module_non_scriptable.py 2022-08-17T13:19:00.1061295Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:00.1073203Z 2022-08-17T13:19:00.1073344Z Running tests... 2022-08-17T13:19:00.1073696Z ---------------------------------------------------------------------- 2022-08-17T13:19:00.1255182Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6457 2022-08-17T13:19:00.1282986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6458 2022-08-17T13:19:01.0787131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:01.0787784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:01.0788571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:01.0789431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:01.0790231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:01.0790816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:01.0791608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:01.0792197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:01.2347444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpe58lye 2022-08-17T13:19:01.2348277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k9hbwl5 2022-08-17T13:19:01.2348951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpe58lye/_remote_module_non_scriptable.py 2022-08-17T13:19:01.2350991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k9hbwl5/_remote_module_non_scriptable.py 2022-08-17T13:19:01.4939828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:01.4940377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:01.8321894Z skip: Need at least 2 CUDA devices (1.725s) 2022-08-17T13:19:01.8322238Z 2022-08-17T13:19:01.8322723Z ---------------------------------------------------------------------- 2022-08-17T13:19:01.8322971Z Ran 1 test in 1.725s 2022-08-17T13:19:01.8323090Z 2022-08-17T13:19:01.8323169Z OK (skipped=1) 2022-08-17T13:19:01.8323283Z 2022-08-17T13:19:01.8323393Z Generating XML reports... 2022-08-17T13:19:01.8357226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131900.xml 2022-08-17T13:19:03.1142447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:03.1143198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:03.1144321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:03.1144919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:03.2657301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v1xtvn6 2022-08-17T13:19:03.2658211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v1xtvn6/_remote_module_non_scriptable.py 2022-08-17T13:19:03.5177086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:03.5188491Z 2022-08-17T13:19:03.5188750Z Running tests... 2022-08-17T13:19:03.5189434Z ---------------------------------------------------------------------- 2022-08-17T13:19:03.5366170Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6490 2022-08-17T13:19:03.5395729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6491 2022-08-17T13:19:04.5042956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:04.5043542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:04.5044675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:04.5045076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:04.5150092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:04.5150491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:04.5152708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:04.5153370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:04.6596731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ll82voo 2022-08-17T13:19:04.6597550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ll82voo/_remote_module_non_scriptable.py 2022-08-17T13:19:04.6710822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcks8jzju 2022-08-17T13:19:04.6712807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcks8jzju/_remote_module_non_scriptable.py 2022-08-17T13:19:04.9160746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:04.9297375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:05.3437358Z skip: Need at least 2 CUDA devices (1.824s) 2022-08-17T13:19:05.3437546Z 2022-08-17T13:19:05.3437877Z ---------------------------------------------------------------------- 2022-08-17T13:19:05.3438158Z Ran 1 test in 1.825s 2022-08-17T13:19:05.3438277Z 2022-08-17T13:19:05.3438356Z OK (skipped=1) 2022-08-17T13:19:05.3438457Z 2022-08-17T13:19:05.3438548Z Generating XML reports... 2022-08-17T13:19:05.3471692Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131903.xml 2022-08-17T13:19:06.6453074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:06.6453479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:06.6455111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:06.6455513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:06.7948728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_jp7vs1 2022-08-17T13:19:06.7950192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_jp7vs1/_remote_module_non_scriptable.py 2022-08-17T13:19:07.0445452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:07.0456998Z 2022-08-17T13:19:07.0457262Z Running tests... 2022-08-17T13:19:07.0457934Z ---------------------------------------------------------------------- 2022-08-17T13:19:07.0633862Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6523 2022-08-17T13:19:07.0661370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6524 2022-08-17T13:19:08.0171503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:08.0172165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:08.0173344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:08.0174119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:08.0399318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:08.0399992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:08.0400805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:08.0401438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:08.1709007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxjngab7 2022-08-17T13:19:08.1710486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxjngab7/_remote_module_non_scriptable.py 2022-08-17T13:19:08.1945125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_w_hc_n 2022-08-17T13:19:08.1946447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_w_hc_n/_remote_module_non_scriptable.py 2022-08-17T13:19:08.4308242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:08.4552339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:08.8701861Z skip: Need at least 2 CUDA devices (1.824s) 2022-08-17T13:19:08.8702091Z 2022-08-17T13:19:08.8702415Z ---------------------------------------------------------------------- 2022-08-17T13:19:08.8702680Z Ran 1 test in 1.824s 2022-08-17T13:19:08.8702874Z 2022-08-17T13:19:08.8702958Z OK (skipped=1) 2022-08-17T13:19:08.8703072Z 2022-08-17T13:19:08.8703164Z Generating XML reports... 2022-08-17T13:19:08.8740110Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131907.xml 2022-08-17T13:19:10.1786361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:10.1786774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:10.1788124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:10.1788514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:10.3284953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxm0t49sz 2022-08-17T13:19:10.3287288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxm0t49sz/_remote_module_non_scriptable.py 2022-08-17T13:19:10.5788123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:10.5798606Z 2022-08-17T13:19:10.5798847Z Running tests... 2022-08-17T13:19:10.5799240Z ---------------------------------------------------------------------- 2022-08-17T13:19:10.5973854Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6556 2022-08-17T13:19:10.6002699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6557 2022-08-17T13:19:11.5446614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:11.5447007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:11.5448845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:11.5449220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:11.5506859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:11.5507233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:11.5510046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:11.5510577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:11.6975657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzwtq7v2 2022-08-17T13:19:11.6976795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzwtq7v2/_remote_module_non_scriptable.py 2022-08-17T13:19:11.7037736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv79kch8 2022-08-17T13:19:11.7039934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv79kch8/_remote_module_non_scriptable.py 2022-08-17T13:19:11.9497484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:11.9547618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:12.3040902Z skip: Need at least 2 CUDA devices (1.724s) 2022-08-17T13:19:12.3041398Z 2022-08-17T13:19:12.3042041Z ---------------------------------------------------------------------- 2022-08-17T13:19:12.3042368Z Ran 1 test in 1.724s 2022-08-17T13:19:12.3042526Z 2022-08-17T13:19:12.3042590Z OK (skipped=1) 2022-08-17T13:19:12.3042703Z 2022-08-17T13:19:12.3042793Z Generating XML reports... 2022-08-17T13:19:12.3076350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131910.xml 2022-08-17T13:19:13.5882990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:13.5883481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:13.5884398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:13.5884937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:13.7358106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirxy9ggp 2022-08-17T13:19:13.7358912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirxy9ggp/_remote_module_non_scriptable.py 2022-08-17T13:19:13.9827404Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:13.9838227Z 2022-08-17T13:19:13.9838623Z Running tests... 2022-08-17T13:19:13.9839075Z ---------------------------------------------------------------------- 2022-08-17T13:19:14.0015153Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6589 2022-08-17T13:19:14.0044039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6590 2022-08-17T13:19:14.9568518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:14.9568948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:14.9571021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:14.9571402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:14.9598021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:14.9598382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:14.9601714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:14.9602091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:15.1109818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxng9gf_u 2022-08-17T13:19:15.1111026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxng9gf_u/_remote_module_non_scriptable.py 2022-08-17T13:19:15.1138632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_3m99cv 2022-08-17T13:19:15.1140455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_3m99cv/_remote_module_non_scriptable.py 2022-08-17T13:19:15.3637940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:15.3672380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:15.7081369Z skip: Need at least 2 CUDA devices (1.724s) 2022-08-17T13:19:15.7081670Z 2022-08-17T13:19:15.7082146Z ---------------------------------------------------------------------- 2022-08-17T13:19:15.7082439Z Ran 1 test in 1.724s 2022-08-17T13:19:15.7082585Z 2022-08-17T13:19:15.7082719Z OK (skipped=1) 2022-08-17T13:19:15.7082834Z 2022-08-17T13:19:15.7082911Z Generating XML reports... 2022-08-17T13:19:15.7117771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131913.xml 2022-08-17T13:19:17.0269903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:17.0270558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:17.0271399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:17.0271979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:17.1801022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxccc7piv 2022-08-17T13:19:17.1802027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxccc7piv/_remote_module_non_scriptable.py 2022-08-17T13:19:17.4322989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:17.4334113Z 2022-08-17T13:19:17.4334599Z Running tests... 2022-08-17T13:19:17.4335030Z ---------------------------------------------------------------------- 2022-08-17T13:19:17.4521427Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6622 2022-08-17T13:19:17.4550544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6623 2022-08-17T13:19:18.4263460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:18.4264121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:18.4265277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:18.4265705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:18.4330586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:18.4330948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:18.4333964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:18.4334346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:18.5808430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe87mzb5l 2022-08-17T13:19:18.5809733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe87mzb5l/_remote_module_non_scriptable.py 2022-08-17T13:19:18.5882790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9opiip7 2022-08-17T13:19:18.5884401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9opiip7/_remote_module_non_scriptable.py 2022-08-17T13:19:18.8344545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:18.8459841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:19.2590071Z skip: Need at least 2 CUDA devices (1.825s) 2022-08-17T13:19:19.2590333Z 2022-08-17T13:19:19.2590658Z ---------------------------------------------------------------------- 2022-08-17T13:19:19.2590909Z Ran 1 test in 1.826s 2022-08-17T13:19:19.2591033Z 2022-08-17T13:19:19.2591111Z OK (skipped=1) 2022-08-17T13:19:19.2591229Z 2022-08-17T13:19:19.2591319Z Generating XML reports... 2022-08-17T13:19:19.2625414Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131917.xml 2022-08-17T13:19:20.5769289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:20.5769770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:20.5771432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:20.5771808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:20.7285001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny9ercx4 2022-08-17T13:19:20.7286180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny9ercx4/_remote_module_non_scriptable.py 2022-08-17T13:19:20.9824400Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:20.9835197Z 2022-08-17T13:19:20.9835285Z Running tests... 2022-08-17T13:19:20.9836203Z ---------------------------------------------------------------------- 2022-08-17T13:19:21.0017662Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6655 2022-08-17T13:19:21.0046141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6656 2022-08-17T13:19:21.9760300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:21.9760958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:21.9762149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:21.9762765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:22.0125207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:22.0126058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:22.0127366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:22.0128009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:22.1321848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8q1u_4f 2022-08-17T13:19:22.1322841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8q1u_4f/_remote_module_non_scriptable.py 2022-08-17T13:19:22.1655118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bfg1ydy 2022-08-17T13:19:22.1656281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bfg1ydy/_remote_module_non_scriptable.py 2022-08-17T13:19:22.3891741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:22.4181734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:22.8085811Z skip: Need at least 2 CUDA devices (1.825s) 2022-08-17T13:19:22.8086263Z 2022-08-17T13:19:22.8086814Z ---------------------------------------------------------------------- 2022-08-17T13:19:22.8087348Z Ran 1 test in 1.825s 2022-08-17T13:19:22.8087471Z 2022-08-17T13:19:22.8087549Z OK (skipped=1) 2022-08-17T13:19:22.8087662Z 2022-08-17T13:19:22.8087753Z Generating XML reports... 2022-08-17T13:19:22.8120816Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131920.xml 2022-08-17T13:19:24.1039118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:24.1039733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:24.1040802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:24.1041385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:24.2605534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k4gibvn 2022-08-17T13:19:24.2606995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k4gibvn/_remote_module_non_scriptable.py 2022-08-17T13:19:24.5186266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-08-17T13:19:24.5196627Z 2022-08-17T13:19:24.5196868Z Running tests... 2022-08-17T13:19:24.5197853Z ---------------------------------------------------------------------- 2022-08-17T13:19:24.5387358Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6688 2022-08-17T13:19:24.5415962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6689 2022-08-17T13:19:25.5201661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:25.5202322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:25.5203097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:25.5203718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:25.5204482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:25.5205076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:25.5206157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:25.5206782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:25.6748150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8xvl_b9 2022-08-17T13:19:25.6749095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8xvl_b9/_remote_module_non_scriptable.py 2022-08-17T13:19:25.6753344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe96zja2v 2022-08-17T13:19:25.6755533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe96zja2v/_remote_module_non_scriptable.py 2022-08-17T13:19:25.9307701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:25.9392101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:26.3457703Z skip: Need at least 2 CUDA devices (1.826s) 2022-08-17T13:19:26.3457997Z 2022-08-17T13:19:26.3458377Z ---------------------------------------------------------------------- 2022-08-17T13:19:26.3458638Z Ran 1 test in 1.826s 2022-08-17T13:19:26.3458746Z 2022-08-17T13:19:26.3458824Z OK (skipped=1) 2022-08-17T13:19:26.3458941Z 2022-08-17T13:19:26.3459031Z Generating XML reports... 2022-08-17T13:19:26.3492806Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131924.xml 2022-08-17T13:19:27.0211664Z Running distributed/test_c10d_spawn_nccl ... [2022-08-17 13:19:27.020726] 2022-08-17T13:19:27.0212660Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:19:27.020817] 2022-08-17T13:19:28.1374081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps383f9pm 2022-08-17T13:19:28.1374579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps383f9pm/_remote_module_non_scriptable.py 2022-08-17T13:19:28.3841520Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-08-17T13:19:28.3842313Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:19:28.3842660Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:19:28.3843121Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:19:28.3843598Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-08-17T13:19:28.3843969Z 2022-08-17T13:19:28.3844282Z 2022-08-17T13:19:28.3846156Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]> 2022-08-17T13:19:28.3847573Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3847993Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3848395Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3848819Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3849238Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3849643Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3850030Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3850443Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:28.3850891Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) 2022-08-17T13:19:29.3236117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:29.3236576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:29.3238438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:29.3238823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:29.4719549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptldzg4yi 2022-08-17T13:19:29.4720919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptldzg4yi/_remote_module_non_scriptable.py 2022-08-17T13:19:29.7213933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:29.7224677Z 2022-08-17T13:19:29.7224792Z Running tests... 2022-08-17T13:19:29.7225380Z ---------------------------------------------------------------------- 2022-08-17T13:19:29.7231181Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.001s) 2022-08-17T13:19:29.7231612Z 2022-08-17T13:19:29.7232358Z ---------------------------------------------------------------------- 2022-08-17T13:19:29.7232676Z Ran 1 test in 0.001s 2022-08-17T13:19:29.7232800Z 2022-08-17T13:19:29.7232877Z OK (skipped=1) 2022-08-17T13:19:29.7232994Z 2022-08-17T13:19:29.7233085Z Generating XML reports... 2022-08-17T13:19:29.7257586Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131929.xml 2022-08-17T13:19:30.9465626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:30.9466021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:30.9467822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:30.9468389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:31.0995742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccq416is 2022-08-17T13:19:31.0996585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccq416is/_remote_module_non_scriptable.py 2022-08-17T13:19:31.3495404Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:31.3506454Z 2022-08-17T13:19:31.3507089Z Running tests... 2022-08-17T13:19:31.3507527Z ---------------------------------------------------------------------- 2022-08-17T13:19:31.3512753Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:19:31.3513081Z 2022-08-17T13:19:31.3513332Z ---------------------------------------------------------------------- 2022-08-17T13:19:31.3513672Z Ran 1 test in 0.001s 2022-08-17T13:19:31.3513795Z 2022-08-17T13:19:31.3513930Z OK (skipped=1) 2022-08-17T13:19:31.3514069Z 2022-08-17T13:19:31.3514165Z Generating XML reports... 2022-08-17T13:19:31.3538882Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131931.xml 2022-08-17T13:19:32.5753564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:32.5754149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:32.5754923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:32.5755287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:32.7243972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8_69j_m 2022-08-17T13:19:32.7245155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8_69j_m/_remote_module_non_scriptable.py 2022-08-17T13:19:32.9723496Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:32.9734847Z 2022-08-17T13:19:32.9734990Z Running tests... 2022-08-17T13:19:32.9735397Z ---------------------------------------------------------------------- 2022-08-17T13:19:32.9741068Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:19:32.9741503Z 2022-08-17T13:19:32.9741962Z ---------------------------------------------------------------------- 2022-08-17T13:19:32.9742394Z Ran 1 test in 0.001s 2022-08-17T13:19:32.9742593Z 2022-08-17T13:19:32.9742721Z OK (skipped=1) 2022-08-17T13:19:32.9742995Z 2022-08-17T13:19:32.9743132Z Generating XML reports... 2022-08-17T13:19:32.9768057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131932.xml 2022-08-17T13:19:34.2185202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:34.2185652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:34.2186969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:34.2187510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:34.3693587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajjsf2vl 2022-08-17T13:19:34.3694701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajjsf2vl/_remote_module_non_scriptable.py 2022-08-17T13:19:34.6174216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:34.6184811Z 2022-08-17T13:19:34.6184941Z Running tests... 2022-08-17T13:19:34.6185591Z ---------------------------------------------------------------------- 2022-08-17T13:19:34.6190690Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-08-17T13:19:34.6191055Z 2022-08-17T13:19:34.6191492Z ---------------------------------------------------------------------- 2022-08-17T13:19:34.6191987Z Ran 1 test in 0.001s 2022-08-17T13:19:34.6192208Z 2022-08-17T13:19:34.6192327Z OK (skipped=1) 2022-08-17T13:19:34.6192442Z 2022-08-17T13:19:34.6192519Z Generating XML reports... 2022-08-17T13:19:34.6217142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131934.xml 2022-08-17T13:19:35.8443197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:35.8443585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:35.8445181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:35.8445562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:35.9944697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uks9ney 2022-08-17T13:19:35.9945914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uks9ney/_remote_module_non_scriptable.py 2022-08-17T13:19:36.2552374Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:36.2562939Z 2022-08-17T13:19:36.2563038Z Running tests... 2022-08-17T13:19:36.2563874Z ---------------------------------------------------------------------- 2022-08-17T13:19:36.2568237Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:36.2568633Z 2022-08-17T13:19:36.2569055Z ---------------------------------------------------------------------- 2022-08-17T13:19:36.2569528Z Ran 1 test in 0.001s 2022-08-17T13:19:36.2569664Z 2022-08-17T13:19:36.2569740Z OK (skipped=1) 2022-08-17T13:19:36.2569857Z 2022-08-17T13:19:36.2569947Z Generating XML reports... 2022-08-17T13:19:36.2596952Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131936.xml 2022-08-17T13:19:37.4877662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:37.4878063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:37.4879592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:37.4880173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:37.6375558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp631li79o 2022-08-17T13:19:37.6376404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp631li79o/_remote_module_non_scriptable.py 2022-08-17T13:19:37.8875460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:37.8885431Z 2022-08-17T13:19:37.8885678Z Running tests... 2022-08-17T13:19:37.8886477Z ---------------------------------------------------------------------- 2022-08-17T13:19:37.8900070Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:19:37.8900349Z 2022-08-17T13:19:37.8900592Z ---------------------------------------------------------------------- 2022-08-17T13:19:37.8900859Z Ran 1 test in 0.001s 2022-08-17T13:19:37.8900985Z 2022-08-17T13:19:37.8901072Z OK (skipped=1) 2022-08-17T13:19:37.8901255Z 2022-08-17T13:19:37.8901344Z Generating XML reports... 2022-08-17T13:19:37.8926703Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131937.xml 2022-08-17T13:19:39.1169736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:39.1170122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:39.1171520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:39.1172114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:39.2694860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf9quxxb 2022-08-17T13:19:39.2696423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf9quxxb/_remote_module_non_scriptable.py 2022-08-17T13:19:39.5204158Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:39.5215031Z 2022-08-17T13:19:39.5215318Z Running tests... 2022-08-17T13:19:39.5215958Z ---------------------------------------------------------------------- 2022-08-17T13:19:39.5219930Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:39.5220358Z 2022-08-17T13:19:39.5220778Z ---------------------------------------------------------------------- 2022-08-17T13:19:39.5221135Z Ran 1 test in 0.000s 2022-08-17T13:19:39.5221252Z 2022-08-17T13:19:39.5221327Z OK (skipped=1) 2022-08-17T13:19:39.5221425Z 2022-08-17T13:19:39.5221513Z Generating XML reports... 2022-08-17T13:19:39.5245525Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131939.xml 2022-08-17T13:19:40.7488892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:40.7489507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:40.7491202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:40.7491801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:40.9013724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez4dt8u_ 2022-08-17T13:19:40.9014666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez4dt8u_/_remote_module_non_scriptable.py 2022-08-17T13:19:41.1538277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:41.1548976Z 2022-08-17T13:19:41.1549426Z Running tests... 2022-08-17T13:19:41.1549857Z ---------------------------------------------------------------------- 2022-08-17T13:19:41.1553914Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:41.1554335Z 2022-08-17T13:19:41.1554641Z ---------------------------------------------------------------------- 2022-08-17T13:19:41.1555094Z Ran 1 test in 0.001s 2022-08-17T13:19:41.1555314Z 2022-08-17T13:19:41.1555436Z OK (skipped=1) 2022-08-17T13:19:41.1555639Z 2022-08-17T13:19:41.1555787Z Generating XML reports... 2022-08-17T13:19:41.1580633Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131941.xml 2022-08-17T13:19:42.4002261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:42.4002663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:42.4004383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:42.4004801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:42.5492001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi2pcxoh 2022-08-17T13:19:42.5493161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi2pcxoh/_remote_module_non_scriptable.py 2022-08-17T13:19:42.7988792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:42.7999955Z 2022-08-17T13:19:42.8000044Z Running tests... 2022-08-17T13:19:42.8000621Z ---------------------------------------------------------------------- 2022-08-17T13:19:42.8004905Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:42.8005323Z 2022-08-17T13:19:42.8005744Z ---------------------------------------------------------------------- 2022-08-17T13:19:42.8006518Z Ran 1 test in 0.001s 2022-08-17T13:19:42.8006642Z 2022-08-17T13:19:42.8006719Z OK (skipped=1) 2022-08-17T13:19:42.8006820Z 2022-08-17T13:19:42.8006909Z Generating XML reports... 2022-08-17T13:19:42.8031027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131942.xml 2022-08-17T13:19:44.0067481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:44.0067860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:44.0069436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:44.0069974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:44.1561634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2q0ru8h 2022-08-17T13:19:44.1562794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2q0ru8h/_remote_module_non_scriptable.py 2022-08-17T13:19:44.4055216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:44.4065989Z 2022-08-17T13:19:44.4066128Z Running tests... 2022-08-17T13:19:44.4066768Z ---------------------------------------------------------------------- 2022-08-17T13:19:44.4071266Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:44.4071779Z 2022-08-17T13:19:44.4072188Z ---------------------------------------------------------------------- 2022-08-17T13:19:44.4072468Z Ran 1 test in 0.001s 2022-08-17T13:19:44.4072591Z 2022-08-17T13:19:44.4072669Z OK (skipped=1) 2022-08-17T13:19:44.4072784Z 2022-08-17T13:19:44.4072860Z Generating XML reports... 2022-08-17T13:19:44.4097185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131944.xml 2022-08-17T13:19:45.6321641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:45.6322135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:45.6323311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:45.6323726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:45.7819810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93n6t3t1 2022-08-17T13:19:45.7821184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93n6t3t1/_remote_module_non_scriptable.py 2022-08-17T13:19:46.0293093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:46.0304230Z 2022-08-17T13:19:46.0304351Z Running tests... 2022-08-17T13:19:46.0304981Z ---------------------------------------------------------------------- 2022-08-17T13:19:46.0309289Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:19:46.0309963Z 2022-08-17T13:19:46.0310343Z ---------------------------------------------------------------------- 2022-08-17T13:19:46.0310803Z Ran 1 test in 0.001s 2022-08-17T13:19:46.0311023Z 2022-08-17T13:19:46.0311161Z OK (skipped=1) 2022-08-17T13:19:46.0311373Z 2022-08-17T13:19:46.0311501Z Generating XML reports... 2022-08-17T13:19:46.0335411Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131946.xml 2022-08-17T13:19:47.2539123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:47.2539626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:47.2541399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:47.2541781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:47.4068278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcruf7zcl 2022-08-17T13:19:47.4069226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcruf7zcl/_remote_module_non_scriptable.py 2022-08-17T13:19:47.6617826Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:47.6629390Z 2022-08-17T13:19:47.6629502Z Running tests... 2022-08-17T13:19:47.6629963Z ---------------------------------------------------------------------- 2022-08-17T13:19:47.6645498Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:19:47.6646113Z 2022-08-17T13:19:47.6646596Z ---------------------------------------------------------------------- 2022-08-17T13:19:47.6646943Z Ran 1 test in 0.002s 2022-08-17T13:19:47.6647077Z 2022-08-17T13:19:47.6647156Z OK (skipped=1) 2022-08-17T13:19:47.6647271Z 2022-08-17T13:19:47.6647367Z Generating XML reports... 2022-08-17T13:19:47.6671442Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131947.xml 2022-08-17T13:19:48.9200175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:48.9201870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:48.9202554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:48.9202930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:49.0718623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphap51hp0 2022-08-17T13:19:49.0719724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphap51hp0/_remote_module_non_scriptable.py 2022-08-17T13:19:49.3219108Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-08-17T13:19:49.3229931Z 2022-08-17T13:19:49.3230197Z Running tests... 2022-08-17T13:19:49.3230829Z ---------------------------------------------------------------------- 2022-08-17T13:19:49.3242411Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-08-17T13:19:49.3242908Z 2022-08-17T13:19:49.3243371Z ---------------------------------------------------------------------- 2022-08-17T13:19:49.3243827Z Ran 1 test in 0.001s 2022-08-17T13:19:49.3244351Z 2022-08-17T13:19:49.3244465Z OK (skipped=1) 2022-08-17T13:19:49.3244664Z 2022-08-17T13:19:49.3244829Z Generating XML reports... 2022-08-17T13:19:49.3268721Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131949.xml 2022-08-17T13:19:49.9463434Z Running distributed/test_data_parallel ... [2022-08-17 13:19:49.945905] 2022-08-17T13:19:49.9464043Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:19:49.945991] 2022-08-17T13:19:51.3204785Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2022-08-17T13:19:51.3219180Z 2022-08-17T13:19:51.3219350Z Running tests... 2022-08-17T13:19:51.3219815Z ---------------------------------------------------------------------- 2022-08-17T13:19:51.3228923Z test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3240474Z test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3249431Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3258183Z test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s) 2022-08-17T13:19:51.3267222Z test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3277703Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3282094Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3313284Z test_data_parallel_model_device (__main__.TestDataParallel) 2022-08-17T13:19:51.3313726Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s) 2022-08-17T13:19:51.3321104Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3330144Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3354180Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s) 2022-08-17T13:19:51.3361532Z test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3377021Z test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3383456Z test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3395972Z test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3400968Z test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3412934Z test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3415033Z test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3420455Z test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3423227Z test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3432030Z test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3441420Z test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3447119Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.001s) 2022-08-17T13:19:51.3461994Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3469141Z test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3476381Z test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3482250Z test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3485088Z test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3487786Z test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s) 2022-08-17T13:19:51.3523312Z test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s) 2022-08-17T13:19:51.3529917Z test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s) 2022-08-17T13:19:51.3548166Z test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.002s) 2022-08-17T13:19:51.3558875Z test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3569342Z test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3581840Z test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3594052Z test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3606491Z test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3619316Z test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3632037Z test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3644702Z test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3657643Z test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3670433Z test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3683227Z test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3695985Z test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3708884Z test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3721674Z test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s) 2022-08-17T13:19:51.3722098Z 2022-08-17T13:19:51.3722460Z ---------------------------------------------------------------------- 2022-08-17T13:19:51.3722739Z Ran 46 tests in 0.050s 2022-08-17T13:19:51.3722869Z 2022-08-17T13:19:51.3722947Z OK (skipped=46) 2022-08-17T13:19:51.3723064Z 2022-08-17T13:19:51.3723157Z Generating XML reports... 2022-08-17T13:19:51.3786924Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220817131951.xml 2022-08-17T13:19:51.3803019Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220817131951.xml 2022-08-17T13:19:51.6635967Z Running distributed/test_distributed_spawn ... [2022-08-17 13:19:51.663226] 2022-08-17T13:19:51.6696016Z MPI not available -- MPI backend tests will be skipped 2022-08-17T13:19:51.6697987Z Running distributed tests for the test backend with env init_method 2022-08-17T13:19:51.6702040Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:19:51.670019] 2022-08-17T13:19:52.9557631Z 2022-08-17T13:19:53.1725043Z Running distributed tests for the test backend with file init_method 2022-08-17T13:19:53.1726236Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:19:53.172402] 2022-08-17T13:19:54.4439595Z 2022-08-17T13:19:54.6474900Z Running distributed tests for the gloo backend with env init_method 2022-08-17T13:19:54.6476348Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:19:54.647406] 2022-08-17T13:19:55.9243078Z 2022-08-17T13:19:55.9281268Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-08-17T13:19:55.9309678Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9310086Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9310403Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9310794Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9311147Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9311514Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9311886Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9312273Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9312680Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9313092Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9313531Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9313947Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9314364Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9314754Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9315179Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9315562Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9315912Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9316245Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9316577Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9316923Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9317286Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9317660Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9317974Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9318266Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9318600Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9318926Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9319249Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9319564Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9319882Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9320176Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9320465Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9320773Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9321075Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9321360Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9321677Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9322005Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9322328Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9322646Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9322995Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9323339Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9323674Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9324006Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9324375Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9324712Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9325032Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9325357Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9325697Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9326226Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9326550Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9326872Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9327188Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9327517Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9327838Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9328157Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9328463Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9328831Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9329135Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9329430Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9329734Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9330033Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9330324Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9330608Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9330923Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9331240Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9331529Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9331828Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9332120Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9332412Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9332715Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9333022Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9333339Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9333625Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9333915Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9334204Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9334489Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9334799Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9335108Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9335397Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9335698Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9336007Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9336338Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9336666Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9337011Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9337357Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9337696Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9338087Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9338428Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9338766Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9339090Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9339433Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9339779Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9340122Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9340475Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9340824Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9341174Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9341489Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9341793Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9342115Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9342385Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9342663Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9343040Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9343330Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9343630Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9343923Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9344219Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9344533Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9344842Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9345150Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9345449Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9345783Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9346102Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9346409Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9346729Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9347048Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9347379Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9347694Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9348025Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9348322Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9348596Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9348902Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9349200Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9349488Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9349794Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9350148Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9350544Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9350877Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9351234Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9351557Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9351883Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9352231Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9352597Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9352937Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9353254Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9353595Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9353914Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9354192Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9354500Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9354821Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9355129Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9355499Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9355841Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9356166Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9356474Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9356835Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9357226Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9357655Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9358137Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9358617Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9359094Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9359564Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9360026Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9360492Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9360965Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9361394Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9361764Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9362106Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9362404Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9362696Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9363010Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9363316Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9363641Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9364001Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9364365Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9364730Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9365039Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9365342Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9365657Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9366185Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9366520Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9366843Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9367162Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9367490Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9367825Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9368150Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9368515Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9368834Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9369163Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9369474Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9369785Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9370142Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9370476Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9370759Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9371066Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9371391Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9371701Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9371993Z test_gather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9372271Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9372556Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9372833Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9373129Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9373420Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9373705Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9374001Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9374284Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9374547Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9374842Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9375151Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9375450Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9375721Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9375988Z test_isend (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9376281Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9376576Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9376899Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9377250Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9377635Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9377953Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9378277Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9378613Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9378930Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9379261Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9379586Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9379894Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9380209Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9380530Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9380823Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9381129Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9381489Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9381902Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9382256Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9382605Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9383037Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9383378Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9383709Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9384036Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9384371Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9384696Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9385044Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9385421Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9385806Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9386177Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9386505Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9386808Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9387107Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9387423Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9387720Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9388003Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9388302Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9388606Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9388874Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9389152Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9389446Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9389736Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9390005Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9390288Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9390584Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9390918Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9391208Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9391494Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9391773Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9392061Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9392358Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9392660Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9392939Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9393235Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9393522Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9393796Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9394123Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9394468Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9394788Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9395091Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9395441Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9395771Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9396075Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9396381Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9396704Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9397027Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9397343Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9397656Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9397959Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9398266Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9398561Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9398886Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9399220Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:55.9399564Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:19:56.8679465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:56.8679852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:56.8682504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:56.8682886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:57.2063901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:19:57.2073981Z 2022-08-17T13:19:57.2074081Z Running tests... 2022-08-17T13:19:57.2074890Z ---------------------------------------------------------------------- 2022-08-17T13:19:57.4734879Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6902 2022-08-17T13:19:57.4763293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6903 2022-08-17T13:19:57.4792256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6904 2022-08-17T13:19:58.4744794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:58.4745839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:58.4747706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:58.4748316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:58.4759726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:58.4760161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:58.4764306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:58.4764905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:58.4874525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:19:58.4875249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:19:58.4878619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:19:58.4879001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:19:58.8438219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:19:58.8539789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:19:58.8540490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:19:58.8541262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:19:58.8541913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:19:58.8543000Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:19:58.8548119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:19:58.8548773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:19:58.8549366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:19:59.1839331Z skip: Need at least 2 CUDA devices (1.976s) 2022-08-17T13:19:59.1839601Z 2022-08-17T13:19:59.1840108Z ---------------------------------------------------------------------- 2022-08-17T13:19:59.1840579Z Ran 1 test in 1.976s 2022-08-17T13:19:59.1840772Z 2022-08-17T13:19:59.1840850Z OK (skipped=1) 2022-08-17T13:19:59.1840963Z 2022-08-17T13:19:59.1841058Z Generating XML reports... 2022-08-17T13:19:59.1876119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131957.xml 2022-08-17T13:20:00.4454442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:00.4454840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:00.4456958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:00.4457558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:00.7907868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:00.7917999Z 2022-08-17T13:20:00.7918098Z Running tests... 2022-08-17T13:20:00.7918620Z ---------------------------------------------------------------------- 2022-08-17T13:20:00.7960425Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-08-17T13:20:00.7962020Z 2022-08-17T13:20:00.7962505Z ---------------------------------------------------------------------- 2022-08-17T13:20:00.7963278Z Ran 1 test in 0.004s 2022-08-17T13:20:00.7963403Z 2022-08-17T13:20:00.7963481Z OK (skipped=1) 2022-08-17T13:20:00.7963595Z 2022-08-17T13:20:00.7963685Z Generating XML reports... 2022-08-17T13:20:00.7986977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132000.xml 2022-08-17T13:20:01.9553566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:01.9553965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:01.9556163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:01.9556533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:02.3022413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:02.3033118Z 2022-08-17T13:20:02.3033729Z Running tests... 2022-08-17T13:20:02.3034250Z ---------------------------------------------------------------------- 2022-08-17T13:20:02.5722287Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6965 2022-08-17T13:20:02.5750203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6966 2022-08-17T13:20:02.5778178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6967 2022-08-17T13:20:03.5835743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:03.5836364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:03.5838153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:03.5838798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:03.5977142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:03.5977852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:03.5982237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:03.5982996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:03.6103152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:03.6103780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:03.6106980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:03.6107619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:03.9807277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:03.9908849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:03.9909697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:03.9910206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:03.9911075Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:03.9911839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:03.9916980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:03.9917966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:03.9918763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:04.2827355Z ok (1.979s) 2022-08-17T13:20:04.2827600Z 2022-08-17T13:20:04.2828059Z ---------------------------------------------------------------------- 2022-08-17T13:20:04.2828470Z Ran 1 test in 1.979s 2022-08-17T13:20:04.2828665Z 2022-08-17T13:20:04.2828759Z OK 2022-08-17T13:20:04.2828905Z 2022-08-17T13:20:04.2829047Z Generating XML reports... 2022-08-17T13:20:04.2864992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132002.xml 2022-08-17T13:20:05.5779949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:05.5780592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:05.5783244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:05.9228638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:05.9229576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:05.9238764Z 2022-08-17T13:20:05.9239055Z Running tests... 2022-08-17T13:20:05.9240012Z ---------------------------------------------------------------------- 2022-08-17T13:20:06.1900842Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7018 2022-08-17T13:20:06.1929586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7019 2022-08-17T13:20:06.1957964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7020 2022-08-17T13:20:07.1664059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:07.1664703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:07.1667940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:07.1668590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:07.1718304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:07.1718696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:07.1722159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:07.1722543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:07.1817153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:07.1817519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:07.1821387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:07.1821749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:07.5445153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:07.5503662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:07.5504239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:07.5504895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:07.5505457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:07.5545686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:07.5611694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:07.5612417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:07.6557990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:07.9007810Z skip: CUDA is not available. (1.977s) 2022-08-17T13:20:07.9008131Z 2022-08-17T13:20:07.9008514Z ---------------------------------------------------------------------- 2022-08-17T13:20:07.9008779Z Ran 1 test in 1.977s 2022-08-17T13:20:07.9008898Z 2022-08-17T13:20:07.9008974Z OK (skipped=1) 2022-08-17T13:20:07.9009091Z 2022-08-17T13:20:07.9009168Z Generating XML reports... 2022-08-17T13:20:07.9043478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132005.xml 2022-08-17T13:20:09.2065201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:09.2065615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:09.2067856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:09.2068611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:09.5471166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:09.5481733Z 2022-08-17T13:20:09.5481995Z Running tests... 2022-08-17T13:20:09.5482694Z ---------------------------------------------------------------------- 2022-08-17T13:20:09.8133638Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7071 2022-08-17T13:20:09.8163214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7072 2022-08-17T13:20:09.8192002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7073 2022-08-17T13:20:10.8211290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:10.8211988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:10.8214457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:10.8215106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:10.8377786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:10.8378398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:10.8380461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:10.8381078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:10.8750811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:10.8751504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:10.8752994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:10.8753675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:11.2424880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:11.2525453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:11.2526252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:11.2526934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:11.2527844Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:11.2528758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:11.2534405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:11.2535024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:11.2535965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:11.2608210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qp2_9cx 2022-08-17T13:20:11.2609629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0ryu20n 2022-08-17T13:20:11.2610303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanumlsm2 2022-08-17T13:20:11.2611137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qp2_9cx/_remote_module_non_scriptable.py 2022-08-17T13:20:11.2611852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0ryu20n/_remote_module_non_scriptable.py 2022-08-17T13:20:11.2612671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanumlsm2/_remote_module_non_scriptable.py 2022-08-17T13:20:11.2778219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:11.2778906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:11.2779571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:11.7246387Z ok (2.176s) 2022-08-17T13:20:11.7246646Z 2022-08-17T13:20:11.7247118Z ---------------------------------------------------------------------- 2022-08-17T13:20:11.7247534Z Ran 1 test in 2.176s 2022-08-17T13:20:11.7247733Z 2022-08-17T13:20:11.7247830Z OK 2022-08-17T13:20:11.7247982Z 2022-08-17T13:20:11.7248130Z Generating XML reports... 2022-08-17T13:20:11.7283370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132009.xml 2022-08-17T13:20:13.0199145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:13.0199552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:13.0201267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:13.0201645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:13.3663212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:13.3672967Z 2022-08-17T13:20:13.3673071Z Running tests... 2022-08-17T13:20:13.3673909Z ---------------------------------------------------------------------- 2022-08-17T13:20:13.6328283Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7124 2022-08-17T13:20:13.6358014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7125 2022-08-17T13:20:13.6386697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7126 2022-08-17T13:20:14.6323490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:14.6323926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:14.6326702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:14.6327203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:14.6380026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:14.6381019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:14.6384043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:14.6384713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:14.6800564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:14.6801288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:14.6804699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:14.6805095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:15.0343871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:15.0444975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:15.0445736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:15.0447060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:15.0447609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:15.0448161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:15.0454888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:15.0455445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:15.0526319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlq9lwuw 2022-08-17T13:20:15.0527085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz2qlnpl 2022-08-17T13:20:15.0528475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlq9lwuw/_remote_module_non_scriptable.py 2022-08-17T13:20:15.0529174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz2qlnpl/_remote_module_non_scriptable.py 2022-08-17T13:20:15.1458321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:15.1529503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4esbase9 2022-08-17T13:20:15.1531716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4esbase9/_remote_module_non_scriptable.py 2022-08-17T13:20:15.1690858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:15.1691597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:15.1692253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:20:15.6441521Z ok (2.277s) 2022-08-17T13:20:15.6441758Z 2022-08-17T13:20:15.6442298Z ---------------------------------------------------------------------- 2022-08-17T13:20:15.6442609Z Ran 1 test in 2.277s 2022-08-17T13:20:15.6442731Z 2022-08-17T13:20:15.6442796Z OK 2022-08-17T13:20:15.6442892Z 2022-08-17T13:20:15.6442973Z Generating XML reports... 2022-08-17T13:20:15.6478284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132013.xml 2022-08-17T13:20:16.9509681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:16.9510362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:16.9513276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:16.9513927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:17.2956668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:17.2966459Z 2022-08-17T13:20:17.2966591Z Running tests... 2022-08-17T13:20:17.2967217Z ---------------------------------------------------------------------- 2022-08-17T13:20:17.5623626Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7177 2022-08-17T13:20:17.5652143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7178 2022-08-17T13:20:17.5681485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7179 2022-08-17T13:20:18.5635949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:18.5636632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:18.5638697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:18.5639341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:18.5768303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:18.5769013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:18.5772601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:18.5773222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:18.5800441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:18.5801074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:18.5804511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:18.5804926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:18.9442150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:18.9543189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:18.9543928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:18.9544577Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:18.9545117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:18.9545666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:18.9651257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:19.0556673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:19.0557293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:19.3729111Z skip: CUDA is not available. (2.076s) 2022-08-17T13:20:19.3729357Z 2022-08-17T13:20:19.3729996Z ---------------------------------------------------------------------- 2022-08-17T13:20:19.3730486Z Ran 1 test in 2.076s 2022-08-17T13:20:19.3730654Z 2022-08-17T13:20:19.3730732Z OK (skipped=1) 2022-08-17T13:20:19.3730844Z 2022-08-17T13:20:19.3730937Z Generating XML reports... 2022-08-17T13:20:19.3769226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132017.xml 2022-08-17T13:20:20.6826663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:20.6829289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:20.6829811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:20.6830192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:21.0214467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:21.0224585Z 2022-08-17T13:20:21.0224732Z Running tests... 2022-08-17T13:20:21.0225373Z ---------------------------------------------------------------------- 2022-08-17T13:20:21.2838994Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7230 2022-08-17T13:20:21.2868267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7231 2022-08-17T13:20:21.2896309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7232 2022-08-17T13:20:22.2876549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:22.2877518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:22.2879117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:22.2879748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:22.2909060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:22.2909456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:22.2913364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:22.2914021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:22.3072328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:22.3072953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:22.3075789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:22.3076448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:22.6756259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:22.6756933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:22.6757436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:22.6758103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:22.6758659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:22.6759222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:22.6862372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:22.7769491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:22.7769956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:23.0944248Z skip: CUDA is not available. (2.072s) 2022-08-17T13:20:23.0944488Z 2022-08-17T13:20:23.0945006Z ---------------------------------------------------------------------- 2022-08-17T13:20:23.0945387Z Ran 1 test in 2.072s 2022-08-17T13:20:23.0945726Z 2022-08-17T13:20:23.0945803Z OK (skipped=1) 2022-08-17T13:20:23.0945917Z 2022-08-17T13:20:23.0946014Z Generating XML reports... 2022-08-17T13:20:23.0983718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132021.xml 2022-08-17T13:20:24.4016613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:24.4017003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:24.4019398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:24.4019879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:24.7466594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:24.7476673Z 2022-08-17T13:20:24.7476872Z Running tests... 2022-08-17T13:20:24.7477341Z ---------------------------------------------------------------------- 2022-08-17T13:20:25.0137447Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7283 2022-08-17T13:20:25.0166087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7284 2022-08-17T13:20:25.0194935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7285 2022-08-17T13:20:25.9762387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:25.9763015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:25.9764199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:25.9764816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:25.9766110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:25.9766714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:25.9767784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:25.9768388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:25.9769166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:25.9769712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:25.9773620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:25.9774284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:26.3454382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:26.3555214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:26.3555840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:26.3556484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:26.3557062Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:26.3557615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:26.3563909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:26.3564438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:26.3565607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:26.7242687Z skip: CUDA is not available. (1.976s) 2022-08-17T13:20:26.7243057Z 2022-08-17T13:20:26.7243536Z ---------------------------------------------------------------------- 2022-08-17T13:20:26.7243805Z Ran 1 test in 1.976s 2022-08-17T13:20:26.7243947Z 2022-08-17T13:20:26.7244012Z OK (skipped=1) 2022-08-17T13:20:26.7244127Z 2022-08-17T13:20:26.7244219Z Generating XML reports... 2022-08-17T13:20:26.7281596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132024.xml 2022-08-17T13:20:27.9907777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:27.9908437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:27.9909643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:27.9910271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:28.3397710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:28.3407222Z 2022-08-17T13:20:28.3407379Z Running tests... 2022-08-17T13:20:28.3408384Z ---------------------------------------------------------------------- 2022-08-17T13:20:28.6104561Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7336 2022-08-17T13:20:28.6132660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7337 2022-08-17T13:20:28.6164450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7338 2022-08-17T13:20:29.5770890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:29.5771559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:29.5774718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:29.5775308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:29.6089099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:29.6089628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:29.6092853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:29.6093526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:29.6372625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:29.6373324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:29.6376897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:29.6377492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:29.9992115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:30.0094086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:30.0094739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:30.0095634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:30.0096203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:30.0097118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:30.0103113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:30.0103718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:30.0104341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:30.3214772Z skip: CUDA is not available. (1.980s) 2022-08-17T13:20:30.3215101Z 2022-08-17T13:20:30.3215642Z ---------------------------------------------------------------------- 2022-08-17T13:20:30.3215906Z Ran 1 test in 1.981s 2022-08-17T13:20:30.3216017Z 2022-08-17T13:20:30.3216095Z OK (skipped=1) 2022-08-17T13:20:30.3216213Z 2022-08-17T13:20:30.3216302Z Generating XML reports... 2022-08-17T13:20:30.3250683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132028.xml 2022-08-17T13:20:31.5652438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:31.5652863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:31.5655633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:31.5656014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:31.9052756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:31.9062429Z 2022-08-17T13:20:31.9062530Z Running tests... 2022-08-17T13:20:31.9063544Z ---------------------------------------------------------------------- 2022-08-17T13:20:32.1670109Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7389 2022-08-17T13:20:32.1700745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7390 2022-08-17T13:20:32.1728305Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7391 2022-08-17T13:20:33.1408311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:33.1409001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:33.1411363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:33.1411966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:33.1445056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:33.1445734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:33.1449276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:33.1449779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:33.1763656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:33.1764145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:33.1767506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:33.1768099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:33.5337297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:33.5438188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:33.5438812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:33.5440066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:33.5441011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:33.5441599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:33.5546865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:33.5547466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:33.6451234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:33.9778405Z skip: CUDA is not available. (2.071s) 2022-08-17T13:20:33.9778709Z 2022-08-17T13:20:33.9779199Z ---------------------------------------------------------------------- 2022-08-17T13:20:33.9779639Z Ran 1 test in 2.071s 2022-08-17T13:20:33.9779836Z 2022-08-17T13:20:33.9779962Z OK (skipped=1) 2022-08-17T13:20:33.9780145Z 2022-08-17T13:20:33.9780276Z Generating XML reports... 2022-08-17T13:20:33.9815478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132031.xml 2022-08-17T13:20:35.2578918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:35.2579618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:35.2580857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:35.2581467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:35.5984254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:35.5997715Z 2022-08-17T13:20:35.5998060Z Running tests... 2022-08-17T13:20:35.5998499Z ---------------------------------------------------------------------- 2022-08-17T13:20:35.8681000Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7442 2022-08-17T13:20:35.8709396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7443 2022-08-17T13:20:35.8737279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7444 2022-08-17T13:20:36.8580637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:36.8581256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:36.8606098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:36.8606731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:36.8611066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:36.8611733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:36.8623951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:36.8624595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:36.8629236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:36.8629857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:36.8638896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:36.8639552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:37.2443783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:37.2544512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:37.2545145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:37.2545905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:37.2546775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:37.2547683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:37.2553199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:37.2554486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:37.2554913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:37.5787150Z skip: CUDA is not available. (1.979s) 2022-08-17T13:20:37.5787421Z 2022-08-17T13:20:37.5787743Z ---------------------------------------------------------------------- 2022-08-17T13:20:37.5788243Z Ran 1 test in 1.979s 2022-08-17T13:20:37.5788374Z 2022-08-17T13:20:37.5788452Z OK (skipped=1) 2022-08-17T13:20:37.5788570Z 2022-08-17T13:20:37.5788661Z Generating XML reports... 2022-08-17T13:20:37.5822484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132035.xml 2022-08-17T13:20:38.8951532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:38.8951936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:38.8955521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:38.8955955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:39.2363442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:39.2373685Z 2022-08-17T13:20:39.2373945Z Running tests... 2022-08-17T13:20:39.2374619Z ---------------------------------------------------------------------- 2022-08-17T13:20:39.4986315Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7495 2022-08-17T13:20:39.5015868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7496 2022-08-17T13:20:39.5044419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7497 2022-08-17T13:20:40.4728760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:40.4729181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:40.4732740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:40.4733438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:40.5020693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:40.5021349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:40.5024034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:40.5024709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:40.5050654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:40.5051311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:40.5055657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:40.5056049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:40.8788362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:40.8788860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:40.8789245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:40.8789941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:40.8790568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:40.8791184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:40.8896654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:40.8897283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:40.9801554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:41.3096150Z skip: CUDA is not available. (2.072s) 2022-08-17T13:20:41.3096485Z 2022-08-17T13:20:41.3096991Z ---------------------------------------------------------------------- 2022-08-17T13:20:41.3097284Z Ran 1 test in 2.072s 2022-08-17T13:20:41.3097392Z 2022-08-17T13:20:41.3097472Z OK (skipped=1) 2022-08-17T13:20:41.3097586Z 2022-08-17T13:20:41.3097684Z Generating XML reports... 2022-08-17T13:20:41.3132465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132039.xml 2022-08-17T13:20:42.6061802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:42.6062207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:42.6064748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:42.6065116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:42.9541769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:42.9552782Z 2022-08-17T13:20:42.9553055Z Running tests... 2022-08-17T13:20:43.2368121Z ---------------------------------------------------------------------- 2022-08-17T13:20:43.2368951Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7548 2022-08-17T13:20:43.2396158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7549 2022-08-17T13:20:43.2425471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7550 2022-08-17T13:20:44.2252669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:44.2253310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:44.2255349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:44.2256011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:44.2282732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:44.2283332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:44.2287819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:44.2288751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:44.2509028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:44.2509688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:44.2512486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:44.2513146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:44.6195860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:44.6297208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:44.6298033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:44.6299363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:44.6300354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:44.6301599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:44.6305693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:44.6306162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:44.6307252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:44.9473902Z skip: Need at least 3 CUDA devices (1.992s) 2022-08-17T13:20:44.9474380Z 2022-08-17T13:20:44.9475303Z ---------------------------------------------------------------------- 2022-08-17T13:20:44.9475764Z Ran 1 test in 1.992s 2022-08-17T13:20:44.9475888Z 2022-08-17T13:20:44.9475970Z OK (skipped=1) 2022-08-17T13:20:44.9476084Z 2022-08-17T13:20:44.9476182Z Generating XML reports... 2022-08-17T13:20:44.9510985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132042.xml 2022-08-17T13:20:46.2175924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:46.2176358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:46.2178273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:46.2178791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:46.5571428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:46.5580460Z 2022-08-17T13:20:46.5580756Z Running tests... 2022-08-17T13:20:46.5581414Z ---------------------------------------------------------------------- 2022-08-17T13:20:46.8181446Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7601 2022-08-17T13:20:46.8211409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7602 2022-08-17T13:20:46.8237923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7603 2022-08-17T13:20:47.7980346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:47.7981107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:47.7985251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:47.7985930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:47.8160168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:47.8160858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:47.8163810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:47.8164488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:47.8169364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:47.8169992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:47.8173886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:47.8174558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:48.1887828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:48.1988521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:48.1988993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:48.1989833Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:48.1990407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:48.1990944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:48.2098300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:48.2098817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:48.3001804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:48.7292319Z ok (2.171s) 2022-08-17T13:20:48.7292574Z 2022-08-17T13:20:48.7293102Z ---------------------------------------------------------------------- 2022-08-17T13:20:48.7293456Z Ran 1 test in 2.171s 2022-08-17T13:20:48.7293614Z 2022-08-17T13:20:48.7293681Z OK 2022-08-17T13:20:48.7293777Z 2022-08-17T13:20:48.7293873Z Generating XML reports... 2022-08-17T13:20:48.7329574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132046.xml 2022-08-17T13:20:50.0397294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:50.0397979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:50.0400554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:50.0401167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:50.3870978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:50.3881801Z 2022-08-17T13:20:50.3882255Z Running tests... 2022-08-17T13:20:50.3882919Z ---------------------------------------------------------------------- 2022-08-17T13:20:50.6579860Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7654 2022-08-17T13:20:50.6607586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7655 2022-08-17T13:20:50.6636247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7656 2022-08-17T13:20:51.6531008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:51.6531648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:51.6533300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:51.6533957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:51.6557953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:51.6558522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:51.6563120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:51.6563540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:51.6603706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:51.6604062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:51.6608311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:51.6608934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:52.0275212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:52.0275646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:52.0276037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:52.0276685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:52.0277244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:52.0277781Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:52.0380814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:52.1287575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:52.1288157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:52.4684063Z skip: CUDA is not available. (2.080s) 2022-08-17T13:20:52.4684402Z 2022-08-17T13:20:52.4684910Z ---------------------------------------------------------------------- 2022-08-17T13:20:52.4685176Z Ran 1 test in 2.080s 2022-08-17T13:20:52.4685297Z 2022-08-17T13:20:52.4685361Z OK (skipped=1) 2022-08-17T13:20:52.4685479Z 2022-08-17T13:20:52.4685568Z Generating XML reports... 2022-08-17T13:20:52.4724014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132050.xml 2022-08-17T13:20:53.7763800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:53.7766528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:53.7767313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:53.7767683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:54.1239515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:54.1249755Z 2022-08-17T13:20:54.1250090Z Running tests... 2022-08-17T13:20:54.1250807Z ---------------------------------------------------------------------- 2022-08-17T13:20:54.3958693Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7707 2022-08-17T13:20:54.3987193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7708 2022-08-17T13:20:54.4015968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7709 2022-08-17T13:20:55.3926683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:55.3927333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:55.3929998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:55.3930587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:55.4351852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:55.4352462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:55.4355240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:55.4355868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:55.4465684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:55.4466379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:55.4470939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:55.4471606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:55.8311400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:20:55.8312137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:20:55.8312537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:20:55.8313182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:55.8313743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:55.8314296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:20:55.8419502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:20:55.8420161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:20:55.9324790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:20:56.2065694Z skip: Need at least 3 CUDA devices (2.081s) 2022-08-17T13:20:56.2065983Z 2022-08-17T13:20:56.2066515Z ---------------------------------------------------------------------- 2022-08-17T13:20:56.2066803Z Ran 1 test in 2.081s 2022-08-17T13:20:56.2066912Z 2022-08-17T13:20:56.2067001Z OK (skipped=1) 2022-08-17T13:20:56.2067115Z 2022-08-17T13:20:56.2067208Z Generating XML reports... 2022-08-17T13:20:56.2101542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132054.xml 2022-08-17T13:20:57.4893075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:57.4893474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:57.4896021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:57.4896389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:57.8335398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:57.8345525Z 2022-08-17T13:20:57.8345652Z Running tests... 2022-08-17T13:20:57.8346061Z ---------------------------------------------------------------------- 2022-08-17T13:20:57.8364958Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-08-17T13:20:57.8365271Z 2022-08-17T13:20:57.8365627Z ---------------------------------------------------------------------- 2022-08-17T13:20:57.8366151Z Ran 1 test in 0.002s 2022-08-17T13:20:57.8366270Z 2022-08-17T13:20:57.8366355Z OK (skipped=1) 2022-08-17T13:20:57.8366469Z 2022-08-17T13:20:57.8366562Z Generating XML reports... 2022-08-17T13:20:57.8391446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132057.xml 2022-08-17T13:20:58.9941759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:20:58.9942159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:20:58.9944743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:20:58.9945135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:20:59.3378047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:20:59.3387662Z 2022-08-17T13:20:59.3387767Z Running tests... 2022-08-17T13:20:59.3388840Z ---------------------------------------------------------------------- 2022-08-17T13:20:59.3403404Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-08-17T13:20:59.6096037Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7770 2022-08-17T13:20:59.6128990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7771 2022-08-17T13:20:59.6160301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7772 2022-08-17T13:21:00.6053769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:00.6054458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:00.6056440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:00.6057103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:00.6126098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:00.6126749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:00.6130298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:00.6131029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:00.6210278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:00.6210790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:00.6214263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:00.6214742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:00.9951479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:00.9952207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:00.9952598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:00.9953226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:00.9953780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:00.9954505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:00.9959374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:00.9960527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:00.9960918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:01.0034986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp913fr174 2022-08-17T13:21:01.0036957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp913fr174/_remote_module_non_scriptable.py 2022-08-17T13:21:01.0069054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b3_i3_4 2022-08-17T13:21:01.0069472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2hc618r 2022-08-17T13:21:01.0071594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b3_i3_4/_remote_module_non_scriptable.py 2022-08-17T13:21:01.0072119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2hc618r/_remote_module_non_scriptable.py 2022-08-17T13:21:01.3209039Z ok (1.982s) 2022-08-17T13:21:01.3209314Z 2022-08-17T13:21:01.3210098Z ---------------------------------------------------------------------- 2022-08-17T13:21:01.3210488Z Ran 1 test in 1.982s 2022-08-17T13:21:01.3210612Z 2022-08-17T13:21:01.3210663Z OK 2022-08-17T13:21:01.3210769Z 2022-08-17T13:21:01.3210865Z Generating XML reports... 2022-08-17T13:21:01.3244371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132059.xml 2022-08-17T13:21:02.5930176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:02.5930806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:02.5932506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:02.5933126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:02.9379836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:02.9390176Z 2022-08-17T13:21:02.9390668Z Running tests... 2022-08-17T13:21:02.9391102Z ---------------------------------------------------------------------- 2022-08-17T13:21:02.9409082Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:21:03.2034197Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7823 2022-08-17T13:21:03.2061488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7824 2022-08-17T13:21:03.2089515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7825 2022-08-17T13:21:04.1889129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:04.1889589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:04.1892252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:04.1892713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:04.1900826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:04.1901472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:04.1905837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:04.1906570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:04.2141917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:04.2142859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:04.2145660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:04.2146168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:04.5734720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:04.5835750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:04.5836357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:04.5837006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:04.5837553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:04.5839804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:04.5944433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:04.5945049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:04.6020396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo7v0lud 2022-08-17T13:21:04.6020918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae_da_8c 2022-08-17T13:21:04.6022137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae_da_8c/_remote_module_non_scriptable.py 2022-08-17T13:21:04.6022822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo7v0lud/_remote_module_non_scriptable.py 2022-08-17T13:21:04.6849203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:04.6922399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xh7o3ov 2022-08-17T13:21:04.6924551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xh7o3ov/_remote_module_non_scriptable.py 2022-08-17T13:21:04.7116262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:04.7116977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:04.7117363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:05.0139705Z ok (2.075s) 2022-08-17T13:21:05.0139954Z 2022-08-17T13:21:05.0140487Z ---------------------------------------------------------------------- 2022-08-17T13:21:05.0140762Z Ran 1 test in 2.075s 2022-08-17T13:21:05.0140884Z 2022-08-17T13:21:05.0140948Z OK 2022-08-17T13:21:05.0141043Z 2022-08-17T13:21:05.0141145Z Generating XML reports... 2022-08-17T13:21:05.0176175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132102.xml 2022-08-17T13:21:06.2924933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:06.2925335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:06.2928016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:06.2928383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:06.6327281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:06.6336827Z 2022-08-17T13:21:06.6336971Z Running tests... 2022-08-17T13:21:06.6337582Z ---------------------------------------------------------------------- 2022-08-17T13:21:06.6356997Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:21:06.8990667Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7876 2022-08-17T13:21:06.9019041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7877 2022-08-17T13:21:06.9047507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7878 2022-08-17T13:21:07.8911990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:07.8912405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:07.8915413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:07.8915861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:07.8993018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:07.8993689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:07.8996674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:07.8997346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:07.9254682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:07.9255142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:07.9258276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:07.9258940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:08.3021834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:08.3124064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:08.3124997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:08.3125500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:08.3126198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:08.3127015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:08.3133034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:08.3133613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:08.3208735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw94pajg 2022-08-17T13:21:08.3209397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppjnt4df 2022-08-17T13:21:08.3210144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw94pajg/_remote_module_non_scriptable.py 2022-08-17T13:21:08.3210595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppjnt4df/_remote_module_non_scriptable.py 2022-08-17T13:21:08.4135710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:08.4210531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28656d7o 2022-08-17T13:21:08.4212509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28656d7o/_remote_module_non_scriptable.py 2022-08-17T13:21:08.4420143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:08.4420715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:08.4421328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:21:08.7095377Z ok (2.075s) 2022-08-17T13:21:08.7095605Z 2022-08-17T13:21:08.7096108Z ---------------------------------------------------------------------- 2022-08-17T13:21:08.7096555Z Ran 1 test in 2.076s 2022-08-17T13:21:08.7096678Z 2022-08-17T13:21:08.7096742Z OK 2022-08-17T13:21:08.7096841Z 2022-08-17T13:21:08.7096941Z Generating XML reports... 2022-08-17T13:21:08.7131102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132106.xml 2022-08-17T13:21:09.9860527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:09.9860910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:09.9863274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:09.9863704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:10.3240390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:10.3250581Z 2022-08-17T13:21:10.3250673Z Running tests... 2022-08-17T13:21:10.3251796Z ---------------------------------------------------------------------- 2022-08-17T13:21:10.3265984Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:21:10.5904725Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7929 2022-08-17T13:21:10.5932065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7930 2022-08-17T13:21:10.5959651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7931 2022-08-17T13:21:11.6091101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:11.6091783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:11.6093478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:11.6094163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:11.6104957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:11.6105528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:11.6110254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:11.6110908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:11.6237385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:11.6237786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:11.6240973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:11.6241364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:11.9826884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:11.9865013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:11.9865665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:11.9866329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:11.9866885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:11.9928289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:11.9972448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:11.9973636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:12.0047281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkbdg8s2 2022-08-17T13:21:12.0048654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt5gz_s5 2022-08-17T13:21:12.0049621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkbdg8s2/_remote_module_non_scriptable.py 2022-08-17T13:21:12.0051093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt5gz_s5/_remote_module_non_scriptable.py 2022-08-17T13:21:12.0940305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:12.1016279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mtcsjxf 2022-08-17T13:21:12.1018448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mtcsjxf/_remote_module_non_scriptable.py 2022-08-17T13:21:12.4012085Z ok (2.076s) 2022-08-17T13:21:12.4012320Z 2022-08-17T13:21:12.4013100Z ---------------------------------------------------------------------- 2022-08-17T13:21:12.4013390Z Ran 1 test in 2.076s 2022-08-17T13:21:12.4013511Z 2022-08-17T13:21:12.4013574Z OK 2022-08-17T13:21:12.4013671Z 2022-08-17T13:21:12.4013766Z Generating XML reports... 2022-08-17T13:21:12.4047327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132110.xml 2022-08-17T13:21:13.6800064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:13.6800553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:13.6802164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:13.6802536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:14.0187111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:14.0196968Z 2022-08-17T13:21:14.0197270Z Running tests... 2022-08-17T13:21:14.0197908Z ---------------------------------------------------------------------- 2022-08-17T13:21:14.2860008Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7982 2022-08-17T13:21:14.2888146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7983 2022-08-17T13:21:14.2916499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7984 2022-08-17T13:21:15.3014302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:15.3014909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:15.3017587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:15.3018220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:15.3258110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:15.3258810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:15.3260639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:15.3261321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:15.3825067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:15.3825723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:15.3828533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:15.3829001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:15.7338273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:15.7339009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:15.7339519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:15.7340179Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:15.7340737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:15.7439341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:15.7446965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:15.7448458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:15.7449052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:15.7451884Z STAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7452452Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7454535Z STAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7461467Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7462496Z STAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7462972Z 2022-08-17T13:21:15.7478102Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7485814Z STAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7488347Z STAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7494436Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7495271Z 2022-08-17T13:21:15.7495980Z STAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7504340Z STAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7504688Z 2022-08-17T13:21:15.7505454Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:15.7510384Z STAGE:2022-08-17 13:21:15 7983:7983 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7511199Z STAGE:2022-08-17 13:21:15 7984:7984 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:15.7512027Z STAGE:2022-08-17 13:21:15 7982:7982 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:16.0967065Z ok (2.077s) 2022-08-17T13:21:16.0967347Z 2022-08-17T13:21:16.0967827Z ---------------------------------------------------------------------- 2022-08-17T13:21:16.0968264Z Ran 1 test in 2.077s 2022-08-17T13:21:16.0968684Z 2022-08-17T13:21:16.0968781Z OK 2022-08-17T13:21:16.0968914Z 2022-08-17T13:21:16.0969069Z Generating XML reports... 2022-08-17T13:21:16.1005583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132114.xml 2022-08-17T13:21:17.3706653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:17.3707032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:17.3709116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:17.3709494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:17.7107276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:17.7116845Z 2022-08-17T13:21:17.7124638Z Running tests... 2022-08-17T13:21:17.7125116Z ---------------------------------------------------------------------- 2022-08-17T13:21:17.9746838Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8038 2022-08-17T13:21:17.9775518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8039 2022-08-17T13:21:17.9805080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8040 2022-08-17T13:21:18.9694489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:18.9695161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:18.9697615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:18.9698208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:18.9715407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:18.9715845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:18.9719987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:18.9720582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:19.0300750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:19.0301153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:19.0304699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:19.0305281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:19.4021878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:19.4123780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:19.4124360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:19.4125064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:19.4125689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:19.4126847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:19.4132108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:19.4133344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:19.4134570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:19.4136909Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4140104Z STAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4140900Z STAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4149324Z STAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4149800Z STAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4160478Z STAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4160959Z STAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4163678Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4174920Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4182340Z STAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4182663Z 2022-08-17T13:21:19.4191642Z STAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4192801Z STAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4195915Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4206493Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:19.4213204Z STAGE:2022-08-17 13:21:19 8038:8038 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:19 8040:8040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4214382Z STAGE:2022-08-17 13:21:19 8039:8039 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:19.4214752Z 2022-08-17T13:21:19.8854967Z ok (2.173s) 2022-08-17T13:21:19.8855129Z 2022-08-17T13:21:19.8855530Z ---------------------------------------------------------------------- 2022-08-17T13:21:19.8855814Z Ran 1 test in 2.174s 2022-08-17T13:21:19.8855933Z 2022-08-17T13:21:19.8855984Z OK 2022-08-17T13:21:19.8856079Z 2022-08-17T13:21:19.8856172Z Generating XML reports... 2022-08-17T13:21:19.8891775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132117.xml 2022-08-17T13:21:21.1835253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:21.1835674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:21.1838505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:21.1838920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:21.5274574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:21.5284798Z 2022-08-17T13:21:21.5285234Z Running tests... 2022-08-17T13:21:21.5286025Z ---------------------------------------------------------------------- 2022-08-17T13:21:21.7905850Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8094 2022-08-17T13:21:21.7934861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8095 2022-08-17T13:21:21.7963407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8096 2022-08-17T13:21:22.7691743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:22.7692723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:22.7694054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:22.7694702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:22.7857764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:22.7858397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:22.7861029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:22.7861438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:22.8136662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:22.8137047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:22.8139963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:22.8140611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:23.1856242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:23.1956658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:23.1957374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:23.1958022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:23.1958567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:23.1959145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:23.2064783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:23.2969623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:23.2970160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:23.3280695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:21:23.3281455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:21:23.3282027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:21:23.3282993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:23.3283891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:23.3284748Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:23.3285652Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3286775Z STAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3287522Z STAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3294436Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3294973Z STAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3304437Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3305098Z STAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3306766Z STAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3317032Z STAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3323824Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3324512Z STAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3330545Z STAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3331153Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3333733Z STAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:23.3334120Z 2022-08-17T13:21:23.3340267Z STAGE:2022-08-17 13:21:23 8095:8095 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3340967Z STAGE:2022-08-17 13:21:23 8094:8094 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:23 8096:8096 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:23.3342037Z 2022-08-17T13:21:23.7012662Z ok (2.172s) 2022-08-17T13:21:23.7012892Z 2022-08-17T13:21:23.7013409Z ---------------------------------------------------------------------- 2022-08-17T13:21:23.7013865Z Ran 1 test in 2.173s 2022-08-17T13:21:23.7014081Z 2022-08-17T13:21:23.7014191Z OK 2022-08-17T13:21:23.7014363Z 2022-08-17T13:21:23.7014514Z Generating XML reports... 2022-08-17T13:21:23.7048893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132121.xml 2022-08-17T13:21:24.9842751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:24.9843145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:24.9845456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:24.9846167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:25.3262146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:25.3271789Z 2022-08-17T13:21:25.3271932Z Running tests... 2022-08-17T13:21:25.3272554Z ---------------------------------------------------------------------- 2022-08-17T13:21:25.5962690Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8159 2022-08-17T13:21:25.5991822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8160 2022-08-17T13:21:25.6019690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8161 2022-08-17T13:21:26.5938831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:26.5939532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:26.5942583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:26.5943276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:26.6007060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:26.6007723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:26.6011427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:26.6012097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:26.6234644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:26.6235350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:26.6237006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:26.6237648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:26.9883386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:26.9884094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:26.9884743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:26.9885779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:26.9886818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:26.9887376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:26.9991595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:26.9991999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:27.0199179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:21:27.0199625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:21:27.0895551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:27.0897304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:21:27.0898452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:27.0908012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:27.0908625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:27.0911711Z STAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0912270Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0919240Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:27.0919784Z 2022-08-17T13:21:27.0928731Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0929226Z STAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0934290Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:27.0934785Z 2022-08-17T13:21:27.0940107Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0940808Z STAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:27.0945305Z STAGE:2022-08-17 13:21:27 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:27.0946333Z STAGE:2022-08-17 13:21:27 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:27.5071680Z ok (2.180s) 2022-08-17T13:21:27.5071923Z 2022-08-17T13:21:27.5072490Z ---------------------------------------------------------------------- 2022-08-17T13:21:27.5072889Z Ran 1 test in 2.180s 2022-08-17T13:21:27.5073022Z 2022-08-17T13:21:27.5073075Z OK 2022-08-17T13:21:27.5073173Z 2022-08-17T13:21:27.5073270Z Generating XML reports... 2022-08-17T13:21:27.5108719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132125.xml 2022-08-17T13:21:28.8157840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:28.8158250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:28.8160234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:28.8160622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:29.1601917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:29.1612071Z 2022-08-17T13:21:29.1612209Z Running tests... 2022-08-17T13:21:29.1613099Z ---------------------------------------------------------------------- 2022-08-17T13:21:29.4258562Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8220 2022-08-17T13:21:29.4286576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8221 2022-08-17T13:21:29.4315521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8222 2022-08-17T13:21:30.4572205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:30.4572727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:30.4575003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:30.4575551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:30.4760787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:30.4761331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:30.4765029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:30.4765590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:30.5179694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:30.5180171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:30.5183512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:30.5183876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:30.8727945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:30.8829056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:30.8829681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:30.8830418Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:30.8831294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:30.8832311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:30.8935529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:30.8939654Z STAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9842137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:30.9842763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:30.9847026Z STAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9848010Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9856182Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9857488Z STAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9857953Z 2022-08-17T13:21:30.9865991Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9869447Z STAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9869862Z 2022-08-17T13:21:30.9875806Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9876629Z STAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9876932Z 2022-08-17T13:21:30.9883309Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9885432Z STAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9886303Z STAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:30.9891150Z STAGE:2022-08-17 13:21:30 8221:8221 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9892063Z STAGE:2022-08-17 13:21:30 8222:8222 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:30 8220:8220 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:30.9892364Z 2022-08-17T13:21:31.4371100Z ok (2.276s) 2022-08-17T13:21:31.4371257Z 2022-08-17T13:21:31.4371593Z ---------------------------------------------------------------------- 2022-08-17T13:21:31.4371844Z Ran 1 test in 2.276s 2022-08-17T13:21:31.4371965Z 2022-08-17T13:21:31.4372049Z OK 2022-08-17T13:21:31.4372144Z 2022-08-17T13:21:31.4372241Z Generating XML reports... 2022-08-17T13:21:31.4407446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132129.xml 2022-08-17T13:21:32.7541212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:32.7541606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:32.7544307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:32.7544725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:33.0949274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:33.0958617Z 2022-08-17T13:21:33.0958868Z Running tests... 2022-08-17T13:21:33.0959490Z ---------------------------------------------------------------------- 2022-08-17T13:21:33.3593233Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8276 2022-08-17T13:21:33.3621074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8277 2022-08-17T13:21:33.3649324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8278 2022-08-17T13:21:34.3409899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:34.3410580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:34.3411686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:34.3412291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:34.3516355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:34.3517035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:34.3519663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:34.3520620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:34.3974635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:34.3975315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:34.3978496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:34.3979089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:34.7620079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:34.7721043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:34.7721825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:34.7722495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:34.7723058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:34.7723590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:34.7828989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:34.7835797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:34.7836795Z STAGE:2022-08-17 13:21:34 8277:8277 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:34.7837455Z STAGE:2022-08-17 13:21:34 8278:8278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:34.8734171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:34.8740619Z STAGE:2022-08-17 13:21:34 8276:8276 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:34.8750377Z STAGE:2022-08-17 13:21:34 8276:8276 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:34.8751371Z STAGE:2022-08-17 13:21:34 8277:8277 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:34 8278:8278 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:34.8751832Z 2022-08-17T13:21:35.2699396Z ok (2.174s) 2022-08-17T13:21:35.2699651Z 2022-08-17T13:21:35.2700110Z ---------------------------------------------------------------------- 2022-08-17T13:21:35.2700502Z Ran 1 test in 2.174s 2022-08-17T13:21:35.2700691Z 2022-08-17T13:21:35.2701029Z OK 2022-08-17T13:21:35.2701188Z 2022-08-17T13:21:35.2701338Z Generating XML reports... 2022-08-17T13:21:35.2736861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132133.xml 2022-08-17T13:21:36.5714646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:36.5715281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:36.5717692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:36.5718105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:36.9155858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:36.9165419Z 2022-08-17T13:21:36.9165702Z Running tests... 2022-08-17T13:21:36.9166481Z ---------------------------------------------------------------------- 2022-08-17T13:21:37.1778570Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8332 2022-08-17T13:21:37.1807066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8333 2022-08-17T13:21:37.1835800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8334 2022-08-17T13:21:38.1588048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:38.1588737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:38.1591663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:38.1592334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:38.1595768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:38.1596165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:38.1600939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:38.1601578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:38.1803594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:38.1804242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:38.1806791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:38.1807437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:38.5420472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:38.5521925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:38.5522640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:38.5523578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:38.5524174Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:38.5524708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:38.5531477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:38.5532562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:38.5533197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:38.5536507Z STAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5539021Z STAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5539489Z 2022-08-17T13:21:38.5549751Z STAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:38.5550424Z 2022-08-17T13:21:38.5550431Z 2022-08-17T13:21:38.5568041Z STAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5568690Z STAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5569360Z STAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5574322Z STAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:38.5575186Z STAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:38.5575499Z 2022-08-17T13:21:38.5587890Z STAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5594090Z STAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:38.5594410Z 2022-08-17T13:21:38.5600243Z STAGE:2022-08-17 13:21:38 8332:8332 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:38.5601140Z STAGE:2022-08-17 13:21:38 8334:8334 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:38 8333:8333 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:38.5601445Z 2022-08-17T13:21:38.9888275Z ok (2.072s) 2022-08-17T13:21:38.9888511Z 2022-08-17T13:21:38.9889035Z ---------------------------------------------------------------------- 2022-08-17T13:21:38.9889507Z Ran 1 test in 2.072s 2022-08-17T13:21:38.9889726Z 2022-08-17T13:21:38.9889848Z OK 2022-08-17T13:21:38.9890021Z 2022-08-17T13:21:38.9890166Z Generating XML reports... 2022-08-17T13:21:38.9923681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132136.xml 2022-08-17T13:21:40.2778238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:40.2778911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:40.2781881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:40.2782476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:40.6232502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:40.6242772Z 2022-08-17T13:21:40.6243234Z Running tests... 2022-08-17T13:21:40.6243889Z ---------------------------------------------------------------------- 2022-08-17T13:21:40.6262086Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-08-17T13:21:40.6262512Z 2022-08-17T13:21:40.6262923Z ---------------------------------------------------------------------- 2022-08-17T13:21:40.6263417Z Ran 1 test in 0.002s 2022-08-17T13:21:40.6263613Z 2022-08-17T13:21:40.6263736Z OK (skipped=1) 2022-08-17T13:21:40.6263919Z 2022-08-17T13:21:40.6264053Z Generating XML reports... 2022-08-17T13:21:40.6290287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132140.xml 2022-08-17T13:21:41.7915192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:41.7915650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:41.7918605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:41.7918992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:42.1358897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:42.1368532Z 2022-08-17T13:21:42.1368842Z Running tests... 2022-08-17T13:21:42.1369506Z ---------------------------------------------------------------------- 2022-08-17T13:21:42.1387612Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-08-17T13:21:42.1387991Z 2022-08-17T13:21:42.1388276Z ---------------------------------------------------------------------- 2022-08-17T13:21:42.1388538Z Ran 1 test in 0.002s 2022-08-17T13:21:42.1388682Z 2022-08-17T13:21:42.1388813Z OK (skipped=1) 2022-08-17T13:21:42.1388916Z 2022-08-17T13:21:42.1389208Z Generating XML reports... 2022-08-17T13:21:42.1413450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132142.xml 2022-08-17T13:21:43.3009431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:43.3009833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:43.3012132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:43.3012514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:43.6457654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:43.6467581Z 2022-08-17T13:21:43.6467799Z Running tests... 2022-08-17T13:21:43.6468158Z ---------------------------------------------------------------------- 2022-08-17T13:21:43.9125366Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8408 2022-08-17T13:21:43.9153692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8409 2022-08-17T13:21:43.9182349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8410 2022-08-17T13:21:44.8874476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:44.8874877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:44.8878014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:44.8878575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:44.8890779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:44.8891155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:44.8895839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:44.8896214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:44.8969089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:44.8969557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:44.8973186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:44.8973813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:45.2636978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:45.2699862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:45.2700578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:45.2701247Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:45.2701898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:45.2737682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:45.2808185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:45.2808824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:45.3748015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:45.4020977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:21:45.4123621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:21:45.4124260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:21:45.4125237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:45.4126248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:45.4127099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:45.4128210Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4128651Z 2022-08-17T13:21:45.4129073Z STAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4137091Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4138186Z STAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4138650Z 2022-08-17T13:21:45.4154428Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4160250Z STAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4160736Z 2022-08-17T13:21:45.4166454Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4167106Z STAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4167412Z 2022-08-17T13:21:45.4176760Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4181760Z STAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:45.4182056Z 2022-08-17T13:21:45.4187672Z STAGE:2022-08-17 13:21:45 8408:8408 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4188360Z STAGE:2022-08-17 13:21:45 8409:8409 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.4188825Z STAGE:2022-08-17 13:21:45 8410:8410 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:45.8232969Z ok (2.176s) 2022-08-17T13:21:45.8233269Z 2022-08-17T13:21:45.8233800Z ---------------------------------------------------------------------- 2022-08-17T13:21:45.8234275Z Ran 1 test in 2.176s 2022-08-17T13:21:45.8234490Z 2022-08-17T13:21:45.8234603Z OK 2022-08-17T13:21:45.8234776Z 2022-08-17T13:21:45.8234925Z Generating XML reports... 2022-08-17T13:21:45.8269180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132143.xml 2022-08-17T13:21:47.1041858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:47.1042340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:47.1045033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:47.1045639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:47.4434457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:47.4444201Z 2022-08-17T13:21:47.4444350Z Running tests... 2022-08-17T13:21:47.4444740Z ---------------------------------------------------------------------- 2022-08-17T13:21:47.7045569Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8473 2022-08-17T13:21:47.7073917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8474 2022-08-17T13:21:47.7101692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8475 2022-08-17T13:21:48.6812960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:48.6813635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:48.6816428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:48.6816859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:48.6936078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:48.6936547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:48.6939612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:48.6940017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:48.7520260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:48.7520934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:48.7523908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:48.7524525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:49.1343836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:49.1344668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:49.1345277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:49.1345935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:49.1346668Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:49.1347228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:49.1453197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:49.1453755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:49.1560847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:21:49.1575955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:21:49.2357772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:49.2359343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:21:49.2360062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:49.2369317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:49.2385202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:49.2388561Z STAGE:2022-08-17 13:21:49 8474:8474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:49.2473756Z STAGE:2022-08-17 13:21:49 8475:8475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:49.2480975Z STAGE:2022-08-17 13:21:49 8475:8475 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:21:49 8474:8474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:49.2481318Z 2022-08-17T13:21:49.2497442Z STAGE:2022-08-17 13:21:49 8475:8475 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:21:49 8474:8474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:21:49.2497787Z 2022-08-17T13:21:49.2502220Z STAGE:2022-08-17 13:21:49 8474:8474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:49.2502826Z STAGE:2022-08-17 13:21:49 8475:8475 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:21:49.7152438Z ok (2.270s) 2022-08-17T13:21:49.7152612Z 2022-08-17T13:21:49.7152968Z ---------------------------------------------------------------------- 2022-08-17T13:21:49.7153436Z Ran 1 test in 2.271s 2022-08-17T13:21:49.7153659Z 2022-08-17T13:21:49.7153758Z OK 2022-08-17T13:21:49.7153916Z 2022-08-17T13:21:49.7154014Z Generating XML reports... 2022-08-17T13:21:49.7191275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132147.xml 2022-08-17T13:21:51.0120043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:51.0120460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:51.0122626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:51.0123018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:51.3576076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:51.3585246Z 2022-08-17T13:21:51.3585400Z Running tests... 2022-08-17T13:21:51.3586035Z ---------------------------------------------------------------------- 2022-08-17T13:21:51.3604414Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-08-17T13:21:51.3604876Z 2022-08-17T13:21:51.3605301Z ---------------------------------------------------------------------- 2022-08-17T13:21:51.3605795Z Ran 1 test in 0.002s 2022-08-17T13:21:51.3606146Z 2022-08-17T13:21:51.3606259Z OK (skipped=1) 2022-08-17T13:21:51.3606379Z 2022-08-17T13:21:51.3606472Z Generating XML reports... 2022-08-17T13:21:51.3632192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132151.xml 2022-08-17T13:21:52.5150737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:52.5151129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:52.5153471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:52.5153994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:52.8560025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:52.8569547Z 2022-08-17T13:21:52.8569824Z Running tests... 2022-08-17T13:21:52.8570447Z ---------------------------------------------------------------------- 2022-08-17T13:21:52.8590747Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-08-17T13:21:52.8591268Z 2022-08-17T13:21:52.8591825Z ---------------------------------------------------------------------- 2022-08-17T13:21:52.8592093Z Ran 1 test in 0.002s 2022-08-17T13:21:52.8592201Z 2022-08-17T13:21:52.8592276Z OK (skipped=1) 2022-08-17T13:21:52.8592388Z 2022-08-17T13:21:52.8592479Z Generating XML reports... 2022-08-17T13:21:52.8617731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132152.xml 2022-08-17T13:21:53.9975818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:53.9976207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:53.9978520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:53.9979093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:54.3354527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:54.3364166Z 2022-08-17T13:21:54.3364264Z Running tests... 2022-08-17T13:21:54.3364815Z ---------------------------------------------------------------------- 2022-08-17T13:21:54.5949009Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8554 2022-08-17T13:21:54.5977165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8555 2022-08-17T13:21:54.6006025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8556 2022-08-17T13:21:55.5779476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:55.5779904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:55.5783248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:55.5783937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:55.5798014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:55.5798664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:55.5802187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:55.5802840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:55.6235341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:55.6236209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:55.6238503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:55.6239039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:55.9877114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:55.9881455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:55.9881925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:55.9883011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:55.9883609Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:55.9978642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:55.9990192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:55.9991073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:56.0990032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:56.5055968Z ok (2.169s) 2022-08-17T13:21:56.5056273Z 2022-08-17T13:21:56.5056663Z ---------------------------------------------------------------------- 2022-08-17T13:21:56.5056931Z Ran 1 test in 2.169s 2022-08-17T13:21:56.5057060Z 2022-08-17T13:21:56.5057124Z OK 2022-08-17T13:21:56.5057219Z 2022-08-17T13:21:56.5057301Z Generating XML reports... 2022-08-17T13:21:56.5092012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132154.xml 2022-08-17T13:21:57.7980249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:57.7980694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:57.7982912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:57.7983373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:58.1354140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:21:58.1363931Z 2022-08-17T13:21:58.1364079Z Running tests... 2022-08-17T13:21:58.1364561Z ---------------------------------------------------------------------- 2022-08-17T13:21:58.3961506Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8607 2022-08-17T13:21:58.3989904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8608 2022-08-17T13:21:58.4018614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8609 2022-08-17T13:21:59.3730896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:59.3731578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:59.3733821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:59.3734469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:59.3794963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:59.3798400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:59.3799293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:59.3800264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:59.4035652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:21:59.4036357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:21:59.4039355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:21:59.4039958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:21:59.7655056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:21:59.7756085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:21:59.7756587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:21:59.7757293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:59.7757935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:59.7759055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:21:59.7764065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:21:59.7764707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:21:59.7766856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:21:59.8179160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:21:59.8179781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:21:59.8180425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:21:59.8181403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:59.8182265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:59.8183198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:21:59.8427413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-08-17T13:21:59.8427934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-08-17T13:21:59.8428327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-08-17T13:21:59.8428980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:21:59.8429540Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:21:59.8430427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:21:59.8655118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-08-17T13:21:59.8655869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-08-17T13:21:59.8656416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-08-17T13:21:59.8657209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:21:59.8658093Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:21:59.8658655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:22:00.3068502Z ok (2.170s) 2022-08-17T13:22:00.3068718Z 2022-08-17T13:22:00.3069081Z ---------------------------------------------------------------------- 2022-08-17T13:22:00.3069461Z Ran 1 test in 2.170s 2022-08-17T13:22:00.3069680Z 2022-08-17T13:22:00.3069803Z OK 2022-08-17T13:22:00.3069980Z 2022-08-17T13:22:00.3070122Z Generating XML reports... 2022-08-17T13:22:00.3104755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132158.xml 2022-08-17T13:22:01.6213512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:01.6213911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:01.6216655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:01.6217021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:01.9666833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:01.9675959Z 2022-08-17T13:22:01.9676068Z Running tests... 2022-08-17T13:22:01.9676690Z ---------------------------------------------------------------------- 2022-08-17T13:22:02.2377011Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8696 2022-08-17T13:22:02.2404316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8697 2022-08-17T13:22:02.2432396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8698 2022-08-17T13:22:03.2183840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:03.2184476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:03.2187287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:03.2187944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:03.2201285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:03.2201899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:03.2205562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:03.2206599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:03.2568750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:03.2569441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:03.2573233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:03.2573922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:03.6172774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:03.6265866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:03.6266305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:03.6266960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:03.6267509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:03.6273709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:03.6375063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:03.6375711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:03.7285376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:03.7587473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:03.7689735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:03.7690191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:03.7690910Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:03.7691489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:03.7791167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:03.7793564Z STAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7795990Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7796549Z STAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7805338Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7805734Z 2022-08-17T13:22:03.7806227Z STAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7821002Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7821652Z STAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7821863Z 2022-08-17T13:22:03.7830177Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7830839Z STAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7831302Z STAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7839579Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7844347Z STAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:03.7844896Z 2022-08-17T13:22:03.7850573Z STAGE:2022-08-17 13:22:03 8698:8698 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7851907Z STAGE:2022-08-17 13:22:03 8696:8696 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:03 8697:8697 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:03.7852214Z 2022-08-17T13:22:04.2483106Z ok (2.280s) 2022-08-17T13:22:04.2483339Z 2022-08-17T13:22:04.2483883Z ---------------------------------------------------------------------- 2022-08-17T13:22:04.2484247Z Ran 1 test in 2.281s 2022-08-17T13:22:04.2484368Z 2022-08-17T13:22:04.2484432Z OK 2022-08-17T13:22:04.2484527Z 2022-08-17T13:22:04.2484613Z Generating XML reports... 2022-08-17T13:22:04.2520085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132201.xml 2022-08-17T13:22:05.5334020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:05.5334424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:05.5337068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:05.5337453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:05.8786696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:05.8796546Z 2022-08-17T13:22:05.8796690Z Running tests... 2022-08-17T13:22:05.8797155Z ---------------------------------------------------------------------- 2022-08-17T13:22:06.1412631Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8761 2022-08-17T13:22:06.1441818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8762 2022-08-17T13:22:06.1469458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8763 2022-08-17T13:22:07.1157009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:07.1157672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:07.1159779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:07.1160368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:07.1269696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:07.1270397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:07.1270882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:07.1271242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:07.1273490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:07.1273891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:07.1274357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:07.1274720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:07.4889955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:07.4990544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:07.4991210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:07.4991950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:07.4992503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:07.4993058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:07.5097883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:07.6001404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:07.6002068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:07.6207761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:07.6209342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:07.6209923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:07.6210672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:07.6211216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:07.6309368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:07.6312753Z STAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6315209Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6315702Z STAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6322883Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6323865Z STAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6324176Z 2022-08-17T13:22:07.6338006Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6344940Z STAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6345256Z 2022-08-17T13:22:07.6353540Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6354088Z 2022-08-17T13:22:07.6354561Z STAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6362164Z STAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6362922Z STAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6363572Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:07.6369896Z STAGE:2022-08-17 13:22:07 8762:8762 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6370723Z STAGE:2022-08-17 13:22:07 8761:8761 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:07.6371546Z STAGE:2022-08-17 13:22:07 8763:8763 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:08.0517694Z ok (2.172s) 2022-08-17T13:22:08.0517929Z 2022-08-17T13:22:08.0518461Z ---------------------------------------------------------------------- 2022-08-17T13:22:08.0518849Z Ran 1 test in 2.172s 2022-08-17T13:22:08.0518970Z 2022-08-17T13:22:08.0519035Z OK 2022-08-17T13:22:08.0519131Z 2022-08-17T13:22:08.0519226Z Generating XML reports... 2022-08-17T13:22:08.0554263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132205.xml 2022-08-17T13:22:09.3723972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:09.3724633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:09.3727584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:09.3728192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:09.7140080Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:09.7150607Z 2022-08-17T13:22:09.7151047Z Running tests... 2022-08-17T13:22:09.7151564Z ---------------------------------------------------------------------- 2022-08-17T13:22:09.9825836Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8826 2022-08-17T13:22:09.9854868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8827 2022-08-17T13:22:09.9883362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8828 2022-08-17T13:22:10.9412075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:10.9412734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:10.9415882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:10.9416537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:11.0004318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:11.0005297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:11.0007042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:11.0007657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:11.0091978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:11.0092586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:11.0096748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:11.0097361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:11.3650276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:11.3660166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:11.3660780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:11.3661426Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:11.3661969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:11.3751522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:11.3767777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:11.3768681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:11.4762130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:11.4882328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:11.4984515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:11.4985709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:11.4986154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:11.4986788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:11.4987594Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:11.4989098Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.4989785Z STAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.4990300Z STAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.4997304Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.4998309Z STAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.4998614Z 2022-08-17T13:22:11.5012455Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.5019410Z STAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.5019894Z 2022-08-17T13:22:11.5027023Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.5027985Z STAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.5028297Z 2022-08-17T13:22:11.5036697Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.5040912Z STAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:11.5041252Z 2022-08-17T13:22:11.5048396Z STAGE:2022-08-17 13:22:11 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:11 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.5048928Z 2022-08-17T13:22:11.5049419Z STAGE:2022-08-17 13:22:11 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:11.8938701Z ok (2.178s) 2022-08-17T13:22:11.8938867Z 2022-08-17T13:22:11.8939208Z ---------------------------------------------------------------------- 2022-08-17T13:22:11.8939475Z Ran 1 test in 2.179s 2022-08-17T13:22:11.8939598Z 2022-08-17T13:22:11.8939649Z OK 2022-08-17T13:22:11.8939745Z 2022-08-17T13:22:11.8939843Z Generating XML reports... 2022-08-17T13:22:11.8975979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132209.xml 2022-08-17T13:22:13.1687828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:13.1688566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:13.1691973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:13.1692663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:13.5061155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:13.5071173Z 2022-08-17T13:22:13.5071613Z Running tests... 2022-08-17T13:22:13.5072042Z ---------------------------------------------------------------------- 2022-08-17T13:22:13.7661818Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8891 2022-08-17T13:22:13.7689285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8892 2022-08-17T13:22:13.7717642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8893 2022-08-17T13:22:14.7370653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:14.7404415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:14.7405619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:14.7406422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:14.7407129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:14.7407788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:14.7408340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:14.7408705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:14.7409149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:14.7409660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:14.7411618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:14.7412230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:15.1210738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:15.1311794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:15.1312297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:15.1312948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:15.1313496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:15.1314055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:15.1418985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:15.2323169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:15.2323857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:15.2531578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:15.2632871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:15.2633516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:15.2634427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:15.2635317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:15.2636273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:15.2637591Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2646942Z STAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2647636Z STAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2650487Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:15.2651121Z STAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:15.2651611Z 2022-08-17T13:22:15.2668379Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2678524Z STAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2679064Z 2022-08-17T13:22:15.2686613Z STAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:15.2687551Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:15.2687899Z 2022-08-17T13:22:15.2698206Z STAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2698838Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2699288Z STAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:15.2708695Z STAGE:2022-08-17 13:22:15 8891:8891 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:15 8892:8892 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:15 8893:8893 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:15.2709442Z 2022-08-17T13:22:15.2709451Z 2022-08-17T13:22:15.6767274Z ok (2.169s) 2022-08-17T13:22:15.6767484Z 2022-08-17T13:22:15.6767978Z ---------------------------------------------------------------------- 2022-08-17T13:22:15.6768399Z Ran 1 test in 2.169s 2022-08-17T13:22:15.6768595Z 2022-08-17T13:22:15.6768699Z OK 2022-08-17T13:22:15.6768849Z 2022-08-17T13:22:15.6768989Z Generating XML reports... 2022-08-17T13:22:15.6804375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132213.xml 2022-08-17T13:22:16.9571416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:16.9571822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:16.9573913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:16.9574482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:17.2974185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:17.2983379Z 2022-08-17T13:22:17.2983532Z Running tests... 2022-08-17T13:22:17.2984136Z ---------------------------------------------------------------------- 2022-08-17T13:22:17.5575488Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8956 2022-08-17T13:22:17.5603718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8957 2022-08-17T13:22:17.5632246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8958 2022-08-17T13:22:18.5508497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:18.5509092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:18.5511423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:18.5511975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:18.5537440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:18.5537970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:18.5542102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:18.5542777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:18.5589722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:18.5590112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:18.5594090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:18.5594568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:18.9269990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:18.9371264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:18.9372321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:18.9372805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:18.9373512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:18.9374056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:18.9380249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:18.9380871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:18.9381455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:18.9384872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:18.9587531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:18.9588146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:18.9589068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:18.9590006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:18.9590825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:18.9591831Z STAGE:2022-08-17 13:22:18 8957:8957 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:18.9692858Z STAGE:2022-08-17 13:22:18 8958:8958 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:18.9699338Z STAGE:2022-08-17 13:22:18 8958:8958 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:18.9700114Z STAGE:2022-08-17 13:22:18 8957:8957 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:18.9715253Z STAGE:2022-08-17 13:22:18 8958:8958 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:18.9716045Z STAGE:2022-08-17 13:22:18 8957:8957 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:18.9721630Z STAGE:2022-08-17 13:22:18 8958:8958 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:18.9722090Z STAGE:2022-08-17 13:22:18 8957:8957 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:19.4682120Z ok (2.169s) 2022-08-17T13:22:19.4682357Z 2022-08-17T13:22:19.4682724Z ---------------------------------------------------------------------- 2022-08-17T13:22:19.4682991Z Ran 1 test in 2.170s 2022-08-17T13:22:19.4683373Z 2022-08-17T13:22:19.4683440Z OK 2022-08-17T13:22:19.4683539Z 2022-08-17T13:22:19.4683634Z Generating XML reports... 2022-08-17T13:22:19.4718533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132217.xml 2022-08-17T13:22:20.7731984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:20.7732385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:20.7735166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:20.7735592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:21.1116125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:21.1126173Z 2022-08-17T13:22:21.1126320Z Running tests... 2022-08-17T13:22:21.1127251Z ---------------------------------------------------------------------- 2022-08-17T13:22:21.3779090Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9017 2022-08-17T13:22:21.3808332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9018 2022-08-17T13:22:21.3839435Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9019 2022-08-17T13:22:22.3682899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:22.3683554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:22.3686597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:22.3687069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:22.3687943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:22.3688464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:22.3693202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:22.3693751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:22.3726079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:22.3726750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:22.3730640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:22.3731302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:22.7310716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:22.7412834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:22.7413921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:22.7414573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:22.7415113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:22.7416029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:22.7422085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:22.7422728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:22.7423329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:22.7423871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:22.7629230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:22.7629990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:22.7630739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:22.7631283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:22.7634365Z STAGE:2022-08-17 13:22:22 9019:9019 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:22 9018:9018 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:22.7634685Z 2022-08-17T13:22:22.7642378Z STAGE:2022-08-17 13:22:22 9018:9018 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:22 9019:9019 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:22.7642707Z 2022-08-17T13:22:22.7663837Z STAGE:2022-08-17 13:22:22 9018:9018 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:22 9019:9019 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:22.7664368Z 2022-08-17T13:22:22.7669468Z STAGE:2022-08-17 13:22:22 9019:9019 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:22 9018:9018 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:22.7669870Z 2022-08-17T13:22:22.7728649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:23.1886756Z ok (2.076s) 2022-08-17T13:22:23.1887047Z 2022-08-17T13:22:23.1887481Z ---------------------------------------------------------------------- 2022-08-17T13:22:23.1887767Z Ran 1 test in 2.076s 2022-08-17T13:22:23.1887875Z 2022-08-17T13:22:23.1887941Z OK 2022-08-17T13:22:23.1888034Z 2022-08-17T13:22:23.1888135Z Generating XML reports... 2022-08-17T13:22:23.1922709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132221.xml 2022-08-17T13:22:24.4903125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:24.4903517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:24.4906162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:24.4906534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:24.8330206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:24.8339943Z 2022-08-17T13:22:24.8340108Z Running tests... 2022-08-17T13:22:24.8340989Z ---------------------------------------------------------------------- 2022-08-17T13:22:25.1021215Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9078 2022-08-17T13:22:25.1049715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9079 2022-08-17T13:22:25.1077689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9080 2022-08-17T13:22:26.1369966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:26.1370616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:26.1373601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:26.1374243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:26.1781827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:26.1782458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:26.1785514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:26.1786253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:26.2058764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:26.2059429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:26.2061916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:26.2062589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:26.5675434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:26.5776611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:26.5777310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:26.5778162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:26.5778726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:26.5779264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:26.5883527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:26.6789391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:26.6789803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:26.6792871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:26.6895049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:26.6895975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:26.6896564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:26.6897382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:26.6900521Z STAGE:2022-08-17 13:22:26 9079:9079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:26.6996340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:26.6999585Z STAGE:2022-08-17 13:22:26 9080:9080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:26.7007695Z STAGE:2022-08-17 13:22:26 9079:9079 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:26.7008494Z STAGE:2022-08-17 13:22:26 9080:9080 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:26.7024377Z STAGE:2022-08-17 13:22:26 9080:9080 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:26 9079:9079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:26.7024835Z 2022-08-17T13:22:26.7029718Z STAGE:2022-08-17 13:22:26 9079:9079 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:26 9080:9080 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:26.7030029Z 2022-08-17T13:22:27.1130486Z ok (2.279s) 2022-08-17T13:22:27.1130945Z 2022-08-17T13:22:27.1131476Z ---------------------------------------------------------------------- 2022-08-17T13:22:27.1131940Z Ran 1 test in 2.279s 2022-08-17T13:22:27.1132107Z 2022-08-17T13:22:27.1132175Z OK 2022-08-17T13:22:27.1132274Z 2022-08-17T13:22:27.1132367Z Generating XML reports... 2022-08-17T13:22:27.1166489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132224.xml 2022-08-17T13:22:28.3943692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:28.3944086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:28.3946289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:28.3946797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:28.7324071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:28.7333325Z 2022-08-17T13:22:28.7333451Z Running tests... 2022-08-17T13:22:28.7334098Z ---------------------------------------------------------------------- 2022-08-17T13:22:28.9942138Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9139 2022-08-17T13:22:28.9970547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9140 2022-08-17T13:22:28.9997998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9141 2022-08-17T13:22:30.0385992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:30.0386406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:30.0388650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:30.0389206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:30.1143500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:30.1143865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:30.1147092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:30.1147550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:30.1183020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:30.1183497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:30.1187247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:30.1187663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:30.4733969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:30.4823938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:30.4824725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:30.4825383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:30.4825942Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:30.4835759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:30.4932690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:30.4933296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:30.5038935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:30.5039346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:30.5847118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:30.5850519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:30.5851482Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:30.5852172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:30.5852716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:30.5855710Z STAGE:2022-08-17 13:22:30 9140:9140 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:30.5955980Z STAGE:2022-08-17 13:22:30 9141:9141 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:30.5964252Z STAGE:2022-08-17 13:22:30 9141:9141 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:30.5964744Z STAGE:2022-08-17 13:22:30 9140:9140 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:30.5983332Z STAGE:2022-08-17 13:22:30 9141:9141 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:30 9140:9140 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:30.5983639Z 2022-08-17T13:22:30.5989341Z STAGE:2022-08-17 13:22:30 9141:9141 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:30.5989956Z STAGE:2022-08-17 13:22:30 9140:9140 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:31.0050063Z ok (2.271s) 2022-08-17T13:22:31.0050329Z 2022-08-17T13:22:31.0050804Z ---------------------------------------------------------------------- 2022-08-17T13:22:31.0051205Z Ran 1 test in 2.272s 2022-08-17T13:22:31.0051395Z 2022-08-17T13:22:31.0051498Z OK 2022-08-17T13:22:31.0051673Z 2022-08-17T13:22:31.0051823Z Generating XML reports... 2022-08-17T13:22:31.0087294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132228.xml 2022-08-17T13:22:32.2940952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:32.2941376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:32.2944151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:32.2944531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:32.6374415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:32.6384823Z 2022-08-17T13:22:32.6385001Z Running tests... 2022-08-17T13:22:32.6385362Z ---------------------------------------------------------------------- 2022-08-17T13:22:32.9022220Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9200 2022-08-17T13:22:32.9050325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9201 2022-08-17T13:22:32.9078714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9202 2022-08-17T13:22:33.8864318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:33.8864742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:33.8867692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:33.8868321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:33.8919924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:33.8920304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:33.8924722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:33.8925371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:33.9175600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:33.9176293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:33.9178484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:33.9179130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:34.2766109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:34.2867884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:34.2868560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:34.2869213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:34.2869757Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:34.2870306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:34.2875946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:34.2877072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:34.2877911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:34.2880857Z STAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2881732Z STAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2882653Z STAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2890790Z STAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2891439Z STAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2891757Z 2022-08-17T13:22:34.2906016Z STAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2912254Z STAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2914142Z STAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2922135Z STAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2922460Z 2022-08-17T13:22:34.2922731Z STAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2932349Z STAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2932813Z STAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2933644Z STAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:34.2939607Z STAGE:2022-08-17 13:22:34 9202:9202 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2940683Z STAGE:2022-08-17 13:22:34 9201:9201 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:34 9200:9200 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:34.2940988Z 2022-08-17T13:22:34.7127266Z ok (2.074s) 2022-08-17T13:22:34.7127507Z 2022-08-17T13:22:34.7128051Z ---------------------------------------------------------------------- 2022-08-17T13:22:34.7128316Z Ran 1 test in 2.074s 2022-08-17T13:22:34.7128423Z 2022-08-17T13:22:34.7128492Z OK 2022-08-17T13:22:34.7128587Z 2022-08-17T13:22:34.7128681Z Generating XML reports... 2022-08-17T13:22:34.7162943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132232.xml 2022-08-17T13:22:36.0132209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:36.0132873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:36.0136072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:36.0136700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:36.3611250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:36.3621179Z 2022-08-17T13:22:36.3621281Z Running tests... 2022-08-17T13:22:36.3622011Z ---------------------------------------------------------------------- 2022-08-17T13:22:36.6360249Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9256 2022-08-17T13:22:36.6389820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9257 2022-08-17T13:22:36.6421039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9258 2022-08-17T13:22:37.6422147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:37.6422819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:37.6427072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:37.6427724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:37.6546036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:37.6546626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:37.6549383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:37.6550088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:37.6584813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:37.6585199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:37.6588689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:37.6589174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:38.0244577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:38.0346413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:38.0347311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:38.0348424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:38.0348997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:38.0349557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:38.0355649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:38.0358358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:38.0358980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:38.3466035Z ok (1.984s) 2022-08-17T13:22:38.3466301Z 2022-08-17T13:22:38.3466833Z ---------------------------------------------------------------------- 2022-08-17T13:22:38.3467352Z Ran 1 test in 1.984s 2022-08-17T13:22:38.3467517Z 2022-08-17T13:22:38.3467583Z OK 2022-08-17T13:22:38.3467679Z 2022-08-17T13:22:38.3467761Z Generating XML reports... 2022-08-17T13:22:38.3502799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132236.xml 2022-08-17T13:22:39.6348966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:39.6349363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:39.6352093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:39.6352542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:39.9830078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:39.9839496Z 2022-08-17T13:22:39.9839672Z Running tests... 2022-08-17T13:22:39.9840243Z ---------------------------------------------------------------------- 2022-08-17T13:22:40.2489763Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9309 2022-08-17T13:22:40.2517661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9310 2022-08-17T13:22:40.2546547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9311 2022-08-17T13:22:41.2284078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:41.2284726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:41.2287018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:41.2287427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:41.2388894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:41.2389257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:41.2392976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:41.2393358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:41.2629925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:41.2630569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:41.2633682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:41.2634376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:41.6247868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:41.6348356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:41.6348787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:41.6349443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:41.6350005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:41.6350559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:41.6455673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:41.6459841Z STAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7361511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:41.7362032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:41.7366110Z STAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7367118Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7374113Z STAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7374866Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7375173Z 2022-08-17T13:22:41.7391532Z STAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7396630Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7396930Z 2022-08-17T13:22:41.7405617Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7406889Z STAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7407437Z 2022-08-17T13:22:41.7415427Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7415921Z STAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7416438Z STAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:41.7425191Z STAGE:2022-08-17 13:22:41 9310:9310 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:41 9309:9309 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7426231Z STAGE:2022-08-17 13:22:41 9311:9311 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:41.7426598Z 2022-08-17T13:22:42.1595410Z ok (2.175s) 2022-08-17T13:22:42.1595648Z 2022-08-17T13:22:42.1596117Z ---------------------------------------------------------------------- 2022-08-17T13:22:42.1596387Z Ran 1 test in 2.176s 2022-08-17T13:22:42.1596497Z 2022-08-17T13:22:42.1596562Z OK 2022-08-17T13:22:42.1596657Z 2022-08-17T13:22:42.1596755Z Generating XML reports... 2022-08-17T13:22:42.1631670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132239.xml 2022-08-17T13:22:43.4401322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:43.4402004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:43.4404277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:43.4404663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:43.7826950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:43.7836936Z 2022-08-17T13:22:43.7837314Z Running tests... 2022-08-17T13:22:43.7837722Z ---------------------------------------------------------------------- 2022-08-17T13:22:44.0495672Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9365 2022-08-17T13:22:44.0522661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9366 2022-08-17T13:22:44.0550501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9367 2022-08-17T13:22:45.0469887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:45.0470527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:45.0473760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:45.0474428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:45.0505633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:45.0506254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:45.0509483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:45.0510102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:45.0896878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:45.0897570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:45.0901482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:45.0902056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:45.4547210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:45.4648190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:45.4648824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:45.4649580Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:45.4650423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:45.4651336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:45.4754916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:45.4759275Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5660756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:45.5661192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:45.5664824Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5665413Z STAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5673773Z STAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5674504Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5675255Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5690691Z STAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5691719Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5692728Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5698988Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5699438Z 2022-08-17T13:22:45.5699718Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5708123Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5708756Z STAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5709191Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:45.5717279Z STAGE:2022-08-17 13:22:45 9365:9365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5718097Z STAGE:2022-08-17 13:22:45 9366:9366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.5718565Z STAGE:2022-08-17 13:22:45 9367:9367 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:45.9601556Z ok (2.176s) 2022-08-17T13:22:45.9601728Z 2022-08-17T13:22:45.9602234Z ---------------------------------------------------------------------- 2022-08-17T13:22:45.9602721Z Ran 1 test in 2.176s 2022-08-17T13:22:45.9602844Z 2022-08-17T13:22:45.9602895Z OK 2022-08-17T13:22:45.9602991Z 2022-08-17T13:22:45.9603090Z Generating XML reports... 2022-08-17T13:22:45.9640722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132243.xml 2022-08-17T13:22:47.2551450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:47.2551849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:47.2553939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:47.2554313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:47.6004003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:47.6014143Z 2022-08-17T13:22:47.6014299Z Running tests... 2022-08-17T13:22:47.6014925Z ---------------------------------------------------------------------- 2022-08-17T13:22:47.8648715Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9421 2022-08-17T13:22:47.8678653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9422 2022-08-17T13:22:47.8709383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9423 2022-08-17T13:22:48.8822429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:48.8823224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:48.8826684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:48.8827313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:48.9016929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:48.9017577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:48.9020122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:48.9020532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:48.9031523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:48.9031899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:48.9036163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:48.9036602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:49.2681248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:49.2783204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:49.2784635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:49.2785326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:49.2785860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:49.2786401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:49.2790996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:49.2791646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:49.2792521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:49.2796127Z STAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2797373Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2798117Z STAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2808240Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:49.2808978Z 2022-08-17T13:22:49.2808986Z 2022-08-17T13:22:49.2825747Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2826359Z STAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2826964Z STAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2833795Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:49.2834851Z STAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:49.2835383Z 2022-08-17T13:22:49.2846221Z STAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2846720Z STAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2847236Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:49.2853668Z STAGE:2022-08-17 13:22:49 9422:9422 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:49.2855041Z STAGE:2022-08-17 13:22:49 9423:9423 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:49 9421:9421 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:49.2855398Z 2022-08-17T13:22:49.6759463Z ok (2.074s) 2022-08-17T13:22:49.6759717Z 2022-08-17T13:22:49.6760316Z ---------------------------------------------------------------------- 2022-08-17T13:22:49.6760590Z Ran 1 test in 2.074s 2022-08-17T13:22:49.6760712Z 2022-08-17T13:22:49.6760764Z OK 2022-08-17T13:22:49.6760860Z 2022-08-17T13:22:49.6760955Z Generating XML reports... 2022-08-17T13:22:49.6795500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132247.xml 2022-08-17T13:22:50.9164119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:50.9164804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:50.9167478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:50.9168338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:51.2605572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:51.2615768Z 2022-08-17T13:22:51.2616148Z Running tests... 2022-08-17T13:22:51.2616770Z ---------------------------------------------------------------------- 2022-08-17T13:22:51.5277342Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9477 2022-08-17T13:22:51.5305263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9478 2022-08-17T13:22:51.5333205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9479 2022-08-17T13:22:52.5290195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:52.5290810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:52.5293146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:52.5293715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:52.5381802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:52.5382470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:52.5385893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:52.5386547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:52.5650038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:52.5650707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:52.5653365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:52.5653990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:52.9249731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:52.9351172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:52.9351751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:52.9352747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:52.9353678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:52.9354216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:52.9358578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:52.9360266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:52.9360717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:53.2383651Z ok (1.976s) 2022-08-17T13:22:53.2383855Z 2022-08-17T13:22:53.2384300Z ---------------------------------------------------------------------- 2022-08-17T13:22:53.2384701Z Ran 1 test in 1.977s 2022-08-17T13:22:53.2384857Z 2022-08-17T13:22:53.2384951Z OK 2022-08-17T13:22:53.2385092Z 2022-08-17T13:22:53.2385231Z Generating XML reports... 2022-08-17T13:22:53.2419968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132251.xml 2022-08-17T13:22:54.5135945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:54.5136910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:54.5139434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:54.5140035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:54.8521906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:54.8532404Z 2022-08-17T13:22:54.8532725Z Running tests... 2022-08-17T13:22:54.8533397Z ---------------------------------------------------------------------- 2022-08-17T13:22:55.1128607Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9530 2022-08-17T13:22:55.1155848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9531 2022-08-17T13:22:55.1183710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9532 2022-08-17T13:22:56.0881751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:56.0882144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:56.0884846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:56.0885344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:56.1237938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:56.1238325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:56.1241502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:56.1242065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:56.1288604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:56.1289269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:56.1293130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:56.1293772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:56.4914275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:22:56.5014573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:22:56.5015292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:22:56.5015950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:56.5016574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:56.5017207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:22:56.5123671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:22:56.5124153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:22:56.6028475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:22:56.6135871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:22:56.6238273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:22:56.6238971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:22:56.6239868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:56.6240627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:56.6241347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:22:56.6242159Z STAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6242866Z STAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6342202Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6348855Z STAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:56.6349521Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:56.6349820Z 2022-08-17T13:22:56.6358396Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6359192Z STAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6359802Z STAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6366221Z STAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:56.6366562Z 2022-08-17T13:22:56.6366838Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:56.6372774Z STAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6373278Z 2022-08-17T13:22:56.6373624Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:22:56.6379111Z STAGE:2022-08-17 13:22:56 9531:9531 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:22:56 9532:9532 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:56.6379435Z 2022-08-17T13:22:56.6379687Z STAGE:2022-08-17 13:22:56 9530:9530 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:22:57.0233353Z ok (2.170s) 2022-08-17T13:22:57.0233542Z 2022-08-17T13:22:57.0233919Z ---------------------------------------------------------------------- 2022-08-17T13:22:57.0234239Z Ran 1 test in 2.170s 2022-08-17T13:22:57.0234360Z 2022-08-17T13:22:57.0234425Z OK 2022-08-17T13:22:57.0234518Z 2022-08-17T13:22:57.0234631Z Generating XML reports... 2022-08-17T13:22:57.0270272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132254.xml 2022-08-17T13:22:58.3309787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:58.3310184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:58.3312546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:58.3312984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:58.6710971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:22:58.6721048Z 2022-08-17T13:22:58.6721411Z Running tests... 2022-08-17T13:22:58.6721861Z ---------------------------------------------------------------------- 2022-08-17T13:22:58.9373381Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9595 2022-08-17T13:22:58.9400712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9596 2022-08-17T13:22:58.9428826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9597 2022-08-17T13:22:59.9259518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:59.9260140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:59.9262361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:59.9263133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:59.9394900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:59.9395607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:59.9399074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:59.9399488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:22:59.9569743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:22:59.9570206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:22:59.9574056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:22:59.9574702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:00.3149708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:00.3251086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:00.3251784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:00.3252679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:00.3253276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:00.3254091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:00.3259646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:00.3260491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:00.3261292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:00.3467797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:00.3569598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:00.3570025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:00.3570658Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:00.3571211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:00.3671756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:00.3674990Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3675394Z 2022-08-17T13:23:00.3692469Z STAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3715004Z STAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3715791Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3726256Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3726742Z 2022-08-17T13:23:00.3729002Z STAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3743562Z STAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3761398Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3761878Z 2022-08-17T13:23:00.3762301Z STAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3768939Z STAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3770346Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:00.3770789Z 2022-08-17T13:23:00.3776657Z STAGE:2022-08-17 13:23:00 9595:9595 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3777686Z STAGE:2022-08-17 13:23:00 9597:9597 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:00 9596:9596 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:00.3778185Z 2022-08-17T13:23:00.8478328Z ok (2.175s) 2022-08-17T13:23:00.8478579Z 2022-08-17T13:23:00.8479126Z ---------------------------------------------------------------------- 2022-08-17T13:23:00.8479476Z Ran 1 test in 2.176s 2022-08-17T13:23:00.8479598Z 2022-08-17T13:23:00.8479648Z OK 2022-08-17T13:23:00.8479746Z 2022-08-17T13:23:00.8479841Z Generating XML reports... 2022-08-17T13:23:00.8515353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132258.xml 2022-08-17T13:23:02.1314609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:02.1315227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:02.1317292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:02.1317668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:02.4866572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:02.4876233Z 2022-08-17T13:23:02.4876343Z Running tests... 2022-08-17T13:23:02.4876960Z ---------------------------------------------------------------------- 2022-08-17T13:23:02.7521956Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9660 2022-08-17T13:23:02.7550384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9661 2022-08-17T13:23:02.7579377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9662 2022-08-17T13:23:03.7763953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:03.7764633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:03.7767420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:03.7768087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:03.8290692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:03.8291458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:03.8293402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:03.8293985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:03.8441406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:03.8442175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:03.8445611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:03.8446506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:04.2096374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:04.2196938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:04.2197548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:04.2198208Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:04.2198763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:04.2199321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:04.2306747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:04.2307406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:04.3210106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:04.3421419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:04.3422101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:04.3422696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:04.3423952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:04.3424802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:04.3425375Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:04.3425883Z STAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3426339Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3426773Z STAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3431892Z STAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3432730Z STAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3433432Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3441042Z STAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3442002Z STAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3442969Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3449723Z STAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3450573Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3450954Z 2022-08-17T13:23:04.3455981Z STAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3456769Z STAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3457478Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:04.3463447Z STAGE:2022-08-17 13:23:04 9661:9661 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:04 9662:9662 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.3463975Z 2022-08-17T13:23:04.3464447Z STAGE:2022-08-17 13:23:04 9660:9660 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:04.7633974Z ok (2.275s) 2022-08-17T13:23:04.7634191Z 2022-08-17T13:23:04.7634641Z ---------------------------------------------------------------------- 2022-08-17T13:23:04.7634918Z Ran 1 test in 2.276s 2022-08-17T13:23:04.7635036Z 2022-08-17T13:23:04.7635105Z OK 2022-08-17T13:23:04.7635204Z 2022-08-17T13:23:04.7635307Z Generating XML reports... 2022-08-17T13:23:04.7670137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132302.xml 2022-08-17T13:23:06.0617345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:06.0617783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:06.0620120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:06.0620491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:06.4026391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:06.4036359Z 2022-08-17T13:23:06.4036481Z Running tests... 2022-08-17T13:23:06.4036913Z ---------------------------------------------------------------------- 2022-08-17T13:23:06.6701720Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9725 2022-08-17T13:23:06.6730024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9726 2022-08-17T13:23:06.6758670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9727 2022-08-17T13:23:07.6507934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:07.6508591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:07.6511045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:07.6511675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:07.6551790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:07.6552448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:07.6556008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:07.6556626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:07.6750372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:07.6750778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:07.6754344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:07.6754727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:08.0379681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:08.0479885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:08.0480542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:08.0481507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:08.0482138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:08.0482691Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:08.0588586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:08.1494284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:08.1494785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:08.1601406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:08.1701291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:08.1701875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:08.1702971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:08.1703531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:08.1704093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:08.1706787Z STAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1806584Z STAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1807328Z STAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1813566Z STAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1813890Z 2022-08-17T13:23:08.1814162Z STAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1823412Z STAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1824034Z 2022-08-17T13:23:08.1824041Z 2022-08-17T13:23:08.1830298Z STAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1831361Z STAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1831856Z 2022-08-17T13:23:08.1837747Z STAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1838702Z STAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:08.1839160Z 2022-08-17T13:23:08.1845568Z STAGE:2022-08-17 13:23:08 9726:9726 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1846627Z STAGE:2022-08-17 13:23:08 9727:9727 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.1847442Z STAGE:2022-08-17 13:23:08 9725:9725 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:08.5811171Z ok (2.177s) 2022-08-17T13:23:08.5811450Z 2022-08-17T13:23:08.5811889Z ---------------------------------------------------------------------- 2022-08-17T13:23:08.5812174Z Ran 1 test in 2.177s 2022-08-17T13:23:08.5812297Z 2022-08-17T13:23:08.5812362Z OK 2022-08-17T13:23:08.5812459Z 2022-08-17T13:23:08.5812572Z Generating XML reports... 2022-08-17T13:23:08.5849818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132306.xml 2022-08-17T13:23:09.8815670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:09.8816044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:09.8819106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:09.8819562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:10.2185071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:10.2194613Z 2022-08-17T13:23:10.2194764Z Running tests... 2022-08-17T13:23:10.2195186Z ---------------------------------------------------------------------- 2022-08-17T13:23:10.4799331Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9790 2022-08-17T13:23:10.4826766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9791 2022-08-17T13:23:10.4854822Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9792 2022-08-17T13:23:11.5053575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:11.5054239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:11.5055683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:11.5056522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:11.5125824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:11.5126764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:11.5130559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:11.5131185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:11.5404955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:11.5405596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:11.5408496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:11.5409195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:11.8992346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:11.9092328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:11.9093112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:11.9093929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:11.9094490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:11.9095041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:11.9200015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:11.9200750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:11.9406014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:11.9406694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:12.0105150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:12.0114146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:12.0115037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:12.0115809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:12.0116344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:12.0118129Z STAGE:2022-08-17 13:23:12 9791:9791 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:12 9792:9792 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:12.0118556Z 2022-08-17T13:23:12.0125365Z STAGE:2022-08-17 13:23:12 9791:9791 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:12 9792:9792 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:12.0125669Z 2022-08-17T13:23:12.0134426Z STAGE:2022-08-17 13:23:12 9791:9791 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:12.0135054Z STAGE:2022-08-17 13:23:12 9792:9792 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:12.0139754Z STAGE:2022-08-17 13:23:12 9792:9792 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:12.0140431Z STAGE:2022-08-17 13:23:12 9791:9791 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:12.4909569Z ok (2.271s) 2022-08-17T13:23:12.4909840Z 2022-08-17T13:23:12.4910368Z ---------------------------------------------------------------------- 2022-08-17T13:23:12.4910627Z Ran 1 test in 2.271s 2022-08-17T13:23:12.4910749Z 2022-08-17T13:23:12.4910828Z OK 2022-08-17T13:23:12.4910926Z 2022-08-17T13:23:12.4911019Z Generating XML reports... 2022-08-17T13:23:12.4948676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132310.xml 2022-08-17T13:23:13.8183644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:13.8184114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:13.8185897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:13.8186458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:14.1601427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:14.1611536Z 2022-08-17T13:23:14.1611678Z Running tests... 2022-08-17T13:23:14.1612519Z ---------------------------------------------------------------------- 2022-08-17T13:23:14.4227064Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9851 2022-08-17T13:23:14.4255000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9852 2022-08-17T13:23:14.4282267Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9853 2022-08-17T13:23:15.4246743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:15.4247431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:15.4249278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:15.4249919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:15.4584636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:15.4585032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:15.4588331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:15.4588959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:15.4641528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:15.4641890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:15.4645832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:15.4646572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:15.8253866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:15.8256296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:15.8256989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:15.8257637Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:15.8258183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:15.8355696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:15.8364277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:15.8364701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:15.8571165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:15.8571572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:15.9367176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:15.9369450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:15.9370454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:15.9380325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:15.9381027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:15.9383940Z STAGE:2022-08-17 13:23:15 9852:9852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:15.9484344Z STAGE:2022-08-17 13:23:15 9853:9853 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:15.9490760Z STAGE:2022-08-17 13:23:15 9853:9853 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:15 9852:9852 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:15.9491103Z 2022-08-17T13:23:15.9499377Z STAGE:2022-08-17 13:23:15 9853:9853 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:15.9500009Z STAGE:2022-08-17 13:23:15 9852:9852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:15.9504739Z STAGE:2022-08-17 13:23:15 9852:9852 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:15 9853:9853 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:15.9505076Z 2022-08-17T13:23:16.4337593Z ok (2.272s) 2022-08-17T13:23:16.4337817Z 2022-08-17T13:23:16.4338301Z ---------------------------------------------------------------------- 2022-08-17T13:23:16.4338762Z Ran 1 test in 2.273s 2022-08-17T13:23:16.4338914Z 2022-08-17T13:23:16.4338979Z OK 2022-08-17T13:23:16.4339072Z 2022-08-17T13:23:16.4339166Z Generating XML reports... 2022-08-17T13:23:16.4373473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132314.xml 2022-08-17T13:23:17.7313075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:17.7313469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:17.7315506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:17.7315897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:18.0745921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:18.0755096Z 2022-08-17T13:23:18.0755211Z Running tests... 2022-08-17T13:23:18.0755700Z ---------------------------------------------------------------------- 2022-08-17T13:23:18.3352995Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9912 2022-08-17T13:23:18.3380601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9913 2022-08-17T13:23:18.3409300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9914 2022-08-17T13:23:19.3187051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:19.3187746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:19.3190425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:19.3190954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:19.3618160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:19.3618604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:19.3621332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:19.3621703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:19.3860014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:19.3860688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:19.3863511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:19.3864196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:19.7408607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:19.7409274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:19.7409666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:19.7410297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:19.7410858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:19.7411404Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:19.7515808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:19.8420376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:19.8421147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:19.8422497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:19.8526868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:19.8527568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:19.8528531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:19.8529089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:19.8531316Z STAGE:2022-08-17 13:23:19 9913:9913 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:19.8627172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:19.8631786Z STAGE:2022-08-17 13:23:19 9914:9914 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:19.8640308Z STAGE:2022-08-17 13:23:19 9913:9913 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:19 9914:9914 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:19.8640620Z 2022-08-17T13:23:19.8652932Z STAGE:2022-08-17 13:23:19 9914:9914 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:19 9913:9913 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:19.8653667Z 2022-08-17T13:23:19.8658349Z STAGE:2022-08-17 13:23:19 9914:9914 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:19.8658893Z STAGE:2022-08-17 13:23:19 9913:9913 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:20.3459445Z ok (2.270s) 2022-08-17T13:23:20.3459702Z 2022-08-17T13:23:20.3460249Z ---------------------------------------------------------------------- 2022-08-17T13:23:20.3460671Z Ran 1 test in 2.270s 2022-08-17T13:23:20.3460780Z 2022-08-17T13:23:20.3460844Z OK 2022-08-17T13:23:20.3460939Z 2022-08-17T13:23:20.3461038Z Generating XML reports... 2022-08-17T13:23:20.3495575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132318.xml 2022-08-17T13:23:21.6470248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:21.6470753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:21.6473020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:21.6473508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:21.9890780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:21.9900220Z 2022-08-17T13:23:21.9900365Z Running tests... 2022-08-17T13:23:21.9901109Z ---------------------------------------------------------------------- 2022-08-17T13:23:22.2582102Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9973 2022-08-17T13:23:22.2610419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9974 2022-08-17T13:23:22.2638888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9975 2022-08-17T13:23:23.2492417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:23.2493088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:23.2494770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:23.2495422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:23.2931166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:23.2931572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:23.2934382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:23.2934887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:23.3118421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:23.3118955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:23.3121751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:23.3122130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:23.6769452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:23.6770096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:23.6770495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:23.6771140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:23.6771701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:23.6772490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:23.6877160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:23.6877706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:23.7083343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:23:23.7083739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:23:23.7781422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:23.7782973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:23:23.7783771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:23.7792211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:23.7792968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:23:23.7795579Z STAGE:2022-08-17 13:23:23 9974:9974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:23.7895494Z STAGE:2022-08-17 13:23:23 9975:9975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:23.7901743Z STAGE:2022-08-17 13:23:23 9975:9975 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:23.7902206Z STAGE:2022-08-17 13:23:23 9974:9974 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:23.7911162Z STAGE:2022-08-17 13:23:23 9975:9975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:23.7911644Z STAGE:2022-08-17 13:23:23 9974:9974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:23.7916322Z STAGE:2022-08-17 13:23:23 9974:9974 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:23 9975:9975 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:23.7916638Z 2022-08-17T13:23:24.2693864Z ok (2.279s) 2022-08-17T13:23:24.2694148Z 2022-08-17T13:23:24.2694612Z ---------------------------------------------------------------------- 2022-08-17T13:23:24.2694878Z Ran 1 test in 2.279s 2022-08-17T13:23:24.2695001Z 2022-08-17T13:23:24.2695053Z OK 2022-08-17T13:23:24.2695150Z 2022-08-17T13:23:24.2695244Z Generating XML reports... 2022-08-17T13:23:24.2729167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132321.xml 2022-08-17T13:23:25.5611487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:25.5611943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:25.5614513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:25.5614901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:25.8989813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:25.8999029Z 2022-08-17T13:23:25.8999211Z Running tests... 2022-08-17T13:23:25.8999624Z ---------------------------------------------------------------------- 2022-08-17T13:23:26.1616412Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10034 2022-08-17T13:23:26.1643452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10035 2022-08-17T13:23:26.1672331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10036 2022-08-17T13:23:27.1514971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:27.1515592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:27.1518423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:27.1519043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:27.1583335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:27.1587322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:27.1588250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:27.1588917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:27.1802241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:27.1802909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:27.1805321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:27.1806519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:27.5643875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:27.5745363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:27.5746008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:27.5746671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:27.5747236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:27.5747798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:27.5753261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:27.5754428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:27.5755535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:27.5757741Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5759151Z STAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5759819Z STAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5766560Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5767324Z STAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5768093Z STAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5776645Z STAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5776958Z 2022-08-17T13:23:27.5777229Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5783481Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5784215Z 2022-08-17T13:23:27.5784672Z STAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5789768Z STAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5790280Z STAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5790853Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:27.5797106Z STAGE:2022-08-17 13:23:27 10034:10034 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5798227Z STAGE:2022-08-17 13:23:27 10036:10036 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:27 10035:10035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:27.5798783Z 2022-08-17T13:23:27.9722355Z ok (2.072s) 2022-08-17T13:23:27.9722612Z 2022-08-17T13:23:27.9723294Z ---------------------------------------------------------------------- 2022-08-17T13:23:27.9723643Z Ran 1 test in 2.072s 2022-08-17T13:23:27.9723771Z 2022-08-17T13:23:27.9723836Z OK 2022-08-17T13:23:27.9723937Z 2022-08-17T13:23:27.9724035Z Generating XML reports... 2022-08-17T13:23:27.9759400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132325.xml 2022-08-17T13:23:29.2759267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:29.2759657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:29.2762360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:29.2762739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:29.6213496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:29.6223488Z 2022-08-17T13:23:29.6223632Z Running tests... 2022-08-17T13:23:29.6224267Z ---------------------------------------------------------------------- 2022-08-17T13:23:29.8890281Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10090 2022-08-17T13:23:29.8919581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10091 2022-08-17T13:23:29.8948056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10092 2022-08-17T13:23:30.8770846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:30.8771502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:30.8774805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:30.8775448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:30.8958204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:30.8958872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:30.8961047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:30.8961455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:30.9023803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:30.9024178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:30.9027973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:30.9028360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:31.2660378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:31.2737276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:31.2737759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:31.2738485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:31.2739044Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:31.2761407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:31.2846087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:31.2846512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:31.2850356Z STAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.2850922Z STAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3774087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:31.3777734Z STAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3783339Z STAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3785206Z STAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3785580Z 2022-08-17T13:23:31.3792998Z STAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3798124Z STAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3798450Z 2022-08-17T13:23:31.3805700Z STAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3807188Z STAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3807751Z 2022-08-17T13:23:31.3812309Z STAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3815196Z STAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:31.3815548Z 2022-08-17T13:23:31.3821597Z STAGE:2022-08-17 13:23:31 10092:10092 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3822118Z STAGE:2022-08-17 13:23:31 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.3822592Z STAGE:2022-08-17 13:23:31 10091:10091 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:31.7998299Z ok (2.177s) 2022-08-17T13:23:31.7998518Z 2022-08-17T13:23:31.7998887Z ---------------------------------------------------------------------- 2022-08-17T13:23:31.7999173Z Ran 1 test in 2.177s 2022-08-17T13:23:31.7999294Z 2022-08-17T13:23:31.7999360Z OK 2022-08-17T13:23:31.7999444Z 2022-08-17T13:23:31.7999541Z Generating XML reports... 2022-08-17T13:23:31.8035282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132329.xml 2022-08-17T13:23:33.1070824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:33.1071484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:33.1074311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:33.1074739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:33.4441115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:33.4451210Z 2022-08-17T13:23:33.4451759Z Running tests... 2022-08-17T13:23:33.4452457Z ---------------------------------------------------------------------- 2022-08-17T13:23:33.7095898Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10146 2022-08-17T13:23:33.7123888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10147 2022-08-17T13:23:33.7152302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10148 2022-08-17T13:23:34.6872715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:34.6873112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:34.6876506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:34.6877191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:34.6930273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:34.6930919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:34.6933701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:34.6934272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:34.6951431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:34.6952032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:34.6956336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:34.6956943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:35.0585952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:35.0687340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:35.0687879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:35.0688850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:35.0689680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:35.0690636Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:35.0793981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:35.1700699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:35.1701287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:35.5201095Z skip: CUDA is not available. (2.075s) 2022-08-17T13:23:35.5201392Z 2022-08-17T13:23:35.5201911Z ---------------------------------------------------------------------- 2022-08-17T13:23:35.5202394Z Ran 1 test in 2.075s 2022-08-17T13:23:35.5202606Z 2022-08-17T13:23:35.5202744Z OK (skipped=1) 2022-08-17T13:23:35.5203216Z 2022-08-17T13:23:35.5203316Z Generating XML reports... 2022-08-17T13:23:35.5238174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132333.xml 2022-08-17T13:23:36.8290272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:36.8290972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:36.8293895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:36.8294510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:37.1756538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:37.1767195Z 2022-08-17T13:23:37.1767655Z Running tests... 2022-08-17T13:23:37.1768317Z ---------------------------------------------------------------------- 2022-08-17T13:23:37.4442868Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10199 2022-08-17T13:23:37.4471007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10200 2022-08-17T13:23:37.4500521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10201 2022-08-17T13:23:38.4661446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:38.4662127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:38.4664617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:38.4665306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:38.5095989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:38.5096662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:38.5098760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:38.5099421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:38.5569930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:38.5570615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:38.5574575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:38.5575217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:38.9186599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:38.9287470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:38.9288119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:38.9288939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:38.9289508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:38.9290069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:38.9395021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:39.0301173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:39.0301693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:39.3550805Z skip: CUDA is not available. (2.178s) 2022-08-17T13:23:39.3551098Z 2022-08-17T13:23:39.3551409Z ---------------------------------------------------------------------- 2022-08-17T13:23:39.3551694Z Ran 1 test in 2.178s 2022-08-17T13:23:39.3551881Z 2022-08-17T13:23:39.3551965Z OK (skipped=1) 2022-08-17T13:23:39.3552078Z 2022-08-17T13:23:39.3552189Z Generating XML reports... 2022-08-17T13:23:39.3587912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132337.xml 2022-08-17T13:23:40.6780666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:40.6781084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:40.6783850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:40.6784236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:41.0186919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:41.0197072Z 2022-08-17T13:23:41.0197179Z Running tests... 2022-08-17T13:23:41.0197788Z ---------------------------------------------------------------------- 2022-08-17T13:23:41.2815122Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10252 2022-08-17T13:23:41.2842919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10253 2022-08-17T13:23:41.2872534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10254 2022-08-17T13:23:42.2752654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:42.2753233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:42.2753753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:42.2754173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:42.2754702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:42.2755112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:42.2758465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:42.2758843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:42.2895875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:42.2896239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:42.2900140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:42.2900702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:42.6523437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:42.6574548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:42.6575068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:42.6575719Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:42.6576273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:42.6625095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:42.6682173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:42.6682815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:42.6686999Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.6687565Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7637238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:42.7659954Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7683377Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7684043Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7684681Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7693474Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7694271Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7715424Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7734446Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7734952Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7735420Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7740456Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7741101Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7741553Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:42.7749696Z STAGE:2022-08-17 13:23:42 10252:10252 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7750492Z STAGE:2022-08-17 13:23:42 10254:10254 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:42.7751278Z STAGE:2022-08-17 13:23:42 10253:10253 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:43.1924820Z ok (2.172s) 2022-08-17T13:23:43.1925041Z 2022-08-17T13:23:43.1925588Z ---------------------------------------------------------------------- 2022-08-17T13:23:43.1926126Z Ran 1 test in 2.173s 2022-08-17T13:23:43.1926250Z 2022-08-17T13:23:43.1926300Z OK 2022-08-17T13:23:43.1926398Z 2022-08-17T13:23:43.1926496Z Generating XML reports... 2022-08-17T13:23:43.1960902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132341.xml 2022-08-17T13:23:44.4671340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:44.4671738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:44.4673967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:44.4674451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:44.8070772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:44.8080306Z 2022-08-17T13:23:44.8080719Z Running tests... 2022-08-17T13:23:44.8081187Z ---------------------------------------------------------------------- 2022-08-17T13:23:45.0726124Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10308 2022-08-17T13:23:45.0753761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10309 2022-08-17T13:23:45.0781044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10310 2022-08-17T13:23:46.0473396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:46.0474019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:46.0476321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:46.0476939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:46.0549902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:46.0550527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:46.0553727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:46.0554405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:46.0892638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:46.0893630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:46.0896283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:46.0896887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:46.4484082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:46.4543981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:46.4544470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:46.4545137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:46.4545684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:46.4584843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:46.4651916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:46.4652520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:46.5597228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:46.8830084Z skip: CUDA is not available. (2.075s) 2022-08-17T13:23:46.8830439Z 2022-08-17T13:23:46.8830972Z ---------------------------------------------------------------------- 2022-08-17T13:23:46.8831303Z Ran 1 test in 2.075s 2022-08-17T13:23:46.8831425Z 2022-08-17T13:23:46.8831502Z OK (skipped=1) 2022-08-17T13:23:46.8831602Z 2022-08-17T13:23:46.8831696Z Generating XML reports... 2022-08-17T13:23:46.8887578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132344.xml 2022-08-17T13:23:48.1947769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:48.1948168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:48.1950473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:48.1950855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:48.5376141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:48.5384900Z 2022-08-17T13:23:48.5385127Z Running tests... 2022-08-17T13:23:48.5385796Z ---------------------------------------------------------------------- 2022-08-17T13:23:48.8068697Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10361 2022-08-17T13:23:48.8098324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10362 2022-08-17T13:23:48.8126523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10363 2022-08-17T13:23:49.7884600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:49.7885272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:49.7887584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:49.7888249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:49.8226468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:49.8227115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:49.8230008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:49.8230678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:49.8350107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:49.8354293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:49.8355103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:49.8355740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:50.2039090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:50.2039829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:50.2040225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:50.2040902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:50.2041447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:50.2041995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:50.2146919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:50.2147326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:50.2150772Z STAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.2151278Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3051996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:50.3055146Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3061599Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3062594Z STAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3063239Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3072309Z STAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3074224Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3074676Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3082973Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3083327Z 2022-08-17T13:23:50.3083600Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3089014Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3089619Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3090071Z STAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:50.3097024Z STAGE:2022-08-17 13:23:50 10362:10362 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3097788Z STAGE:2022-08-17 13:23:50 10363:10363 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.3098430Z STAGE:2022-08-17 13:23:50 10361:10361 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:50.7177653Z ok (2.179s) 2022-08-17T13:23:50.7177900Z 2022-08-17T13:23:50.7178432Z ---------------------------------------------------------------------- 2022-08-17T13:23:50.7178815Z Ran 1 test in 2.179s 2022-08-17T13:23:50.7178924Z 2022-08-17T13:23:50.7178995Z OK 2022-08-17T13:23:50.7179092Z 2022-08-17T13:23:50.7179189Z Generating XML reports... 2022-08-17T13:23:50.7216741Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132348.xml 2022-08-17T13:23:52.0309424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:52.0309841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:52.0312516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:52.0312894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:52.3794985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:52.3805569Z 2022-08-17T13:23:52.3806151Z Running tests... 2022-08-17T13:23:52.3806800Z ---------------------------------------------------------------------- 2022-08-17T13:23:52.6445503Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10417 2022-08-17T13:23:52.6474124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10418 2022-08-17T13:23:52.6502816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10419 2022-08-17T13:23:53.6241079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:53.6241468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:53.6243997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:53.6244455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:53.6246987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:53.6247502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:53.6251637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:53.6252086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:53.6284560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:53.6284905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:53.6289524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:53.6290176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:54.0004012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:54.0004464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:54.0004906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:54.0005566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:54.0006378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:54.0006996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:54.0013029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:54.0014245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:54.0015069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:54.0017917Z STAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0018607Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0019510Z STAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0027025Z STAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0027762Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0028382Z STAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0035514Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0036060Z 2022-08-17T13:23:54.0036352Z STAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0042873Z STAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0043223Z 2022-08-17T13:23:54.0043506Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0048751Z STAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0049406Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0049873Z STAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:54.0057413Z STAGE:2022-08-17 13:23:54 10419:10419 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:54 10418:10418 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.0057788Z 2022-08-17T13:23:54.0058063Z STAGE:2022-08-17 13:23:54 10417:10417 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:54.3551604Z ok (1.974s) 2022-08-17T13:23:54.3552054Z 2022-08-17T13:23:54.3552452Z ---------------------------------------------------------------------- 2022-08-17T13:23:54.3552708Z Ran 1 test in 1.975s 2022-08-17T13:23:54.3552831Z 2022-08-17T13:23:54.3552895Z OK 2022-08-17T13:23:54.3552990Z 2022-08-17T13:23:54.3553105Z Generating XML reports... 2022-08-17T13:23:54.3588620Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132352.xml 2022-08-17T13:23:55.6521230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:55.6521607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:55.6523725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:55.6524100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:55.9963967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:55.9973780Z 2022-08-17T13:23:55.9973941Z Running tests... 2022-08-17T13:23:55.9974388Z ---------------------------------------------------------------------- 2022-08-17T13:23:56.2677157Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10473 2022-08-17T13:23:56.2705873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10474 2022-08-17T13:23:56.2734491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10475 2022-08-17T13:23:57.2864498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:57.2864890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:57.2867832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:57.2868388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:57.2902718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:57.2903204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:57.2906811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:57.2907172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:57.3034505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:57.3034995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:57.3038330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:57.3038791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:57.6779364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:23:57.6880891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:23:57.6881619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:23:57.6882274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:57.6882839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:57.6883376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:23:57.6889116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:23:57.6890264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:23:57.6890811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:23:57.6895220Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6895705Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6897255Z STAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6903677Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6904670Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6904985Z 2022-08-17T13:23:57.6914429Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6914740Z 2022-08-17T13:23:57.6915169Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6922661Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6923783Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6924261Z 2022-08-17T13:23:57.6930110Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6930830Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6931690Z STAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:23:57.6938673Z STAGE:2022-08-17 13:23:57 10475:10475 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6939478Z STAGE:2022-08-17 13:23:57 10473:10473 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:23:57 10474:10474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:23:57.6939789Z 2022-08-17T13:23:58.0785400Z ok (2.081s) 2022-08-17T13:23:58.0785678Z 2022-08-17T13:23:58.0786192Z ---------------------------------------------------------------------- 2022-08-17T13:23:58.0786660Z Ran 1 test in 2.081s 2022-08-17T13:23:58.0786872Z 2022-08-17T13:23:58.0786984Z OK 2022-08-17T13:23:58.0787159Z 2022-08-17T13:23:58.0787331Z Generating XML reports... 2022-08-17T13:23:58.0821592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132355.xml 2022-08-17T13:23:59.3859486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:23:59.3859865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:23:59.3862233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:23:59.3862755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:23:59.7279240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:23:59.7289151Z 2022-08-17T13:23:59.7289286Z Running tests... 2022-08-17T13:23:59.7289900Z ---------------------------------------------------------------------- 2022-08-17T13:23:59.9917053Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10529 2022-08-17T13:23:59.9945336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10530 2022-08-17T13:23:59.9974709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10531 2022-08-17T13:24:00.9925365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:00.9926326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:00.9928782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:00.9929397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:00.9951686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:00.9952319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:00.9955880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:00.9956546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:01.0322206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:01.0323208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:01.0326571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:01.0327043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:01.3966693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:24:01.4067523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:24:01.4068156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:24:01.4069194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:01.4070081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:01.4071025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:01.4174972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:24:01.5081601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:24:01.5082150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:24:01.8025047Z skip: CUDA is not available. (2.073s) 2022-08-17T13:24:01.8025402Z 2022-08-17T13:24:01.8025859Z ---------------------------------------------------------------------- 2022-08-17T13:24:01.8026278Z Ran 1 test in 2.073s 2022-08-17T13:24:01.8026481Z 2022-08-17T13:24:01.8026603Z OK (skipped=1) 2022-08-17T13:24:01.8026773Z 2022-08-17T13:24:01.8026886Z Generating XML reports... 2022-08-17T13:24:01.8062381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132359.xml 2022-08-17T13:24:03.0999667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:03.1000350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:03.1003236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:03.1003822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:03.4414211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:03.4424157Z 2022-08-17T13:24:03.4424362Z Running tests... 2022-08-17T13:24:03.4425257Z ---------------------------------------------------------------------- 2022-08-17T13:24:03.7050017Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10582 2022-08-17T13:24:03.7077527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10583 2022-08-17T13:24:03.7105642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10584 2022-08-17T13:24:04.6836497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:04.6837134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:04.6840124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:04.6840750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:04.6890306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:04.6890929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:04.6895305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:04.6896222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:04.6896706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:04.6897081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:04.6899300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:04.6899877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:05.0487354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:24:05.0589282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:24:05.0589792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:24:05.0590447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:05.0591009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:05.0591560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:05.0598768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:24:05.0599456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:24:05.0599813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:24:05.4152315Z skip: CUDA is not available. (1.973s) 2022-08-17T13:24:05.4152655Z 2022-08-17T13:24:05.4153173Z ---------------------------------------------------------------------- 2022-08-17T13:24:05.4153480Z Ran 1 test in 1.973s 2022-08-17T13:24:05.4153620Z 2022-08-17T13:24:05.4153698Z OK (skipped=1) 2022-08-17T13:24:05.4153813Z 2022-08-17T13:24:05.4153904Z Generating XML reports... 2022-08-17T13:24:05.4189369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132403.xml 2022-08-17T13:24:06.7017737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:06.7018180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:06.7020919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:06.7021564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:07.0398731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:07.0409107Z 2022-08-17T13:24:07.0409232Z Running tests... 2022-08-17T13:24:07.0410225Z ---------------------------------------------------------------------- 2022-08-17T13:24:07.3030172Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10635 2022-08-17T13:24:07.3057424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10636 2022-08-17T13:24:07.3084826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10637 2022-08-17T13:24:08.2845109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:08.2845500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:08.2848764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:08.2849243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:08.3283109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:08.3283752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:08.3284388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:08.3284806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:08.3287064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:08.3287439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:08.3287917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:08.3288274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:08.7022475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:24:08.7068931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:24:08.7069391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:24:08.7070052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:08.7070607Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:08.7124551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:08.7177962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:24:08.7178554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:24:08.8137029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:24:09.1132606Z skip: CUDA is not available. (2.072s) 2022-08-17T13:24:09.1132896Z 2022-08-17T13:24:09.1133421Z ---------------------------------------------------------------------- 2022-08-17T13:24:09.1133724Z Ran 1 test in 2.072s 2022-08-17T13:24:09.1133843Z 2022-08-17T13:24:09.1133921Z OK (skipped=1) 2022-08-17T13:24:09.1134032Z 2022-08-17T13:24:09.1134121Z Generating XML reports... 2022-08-17T13:24:09.1168594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132407.xml 2022-08-17T13:24:10.4241659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:10.4242634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:10.4244409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:10.4245049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:10.7644884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:10.7655228Z 2022-08-17T13:24:10.7655595Z Running tests... 2022-08-17T13:24:10.7656254Z ---------------------------------------------------------------------- 2022-08-17T13:24:10.7673271Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:24:10.7673653Z 2022-08-17T13:24:10.7674122Z ---------------------------------------------------------------------- 2022-08-17T13:24:10.7674489Z Ran 1 test in 0.002s 2022-08-17T13:24:10.7674612Z 2022-08-17T13:24:10.7674689Z OK (skipped=1) 2022-08-17T13:24:10.7674801Z 2022-08-17T13:24:10.7674891Z Generating XML reports... 2022-08-17T13:24:10.7698913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132410.xml 2022-08-17T13:24:11.9128836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:11.9129234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:11.9131892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:11.9132277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:12.2550129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:12.2559743Z 2022-08-17T13:24:12.2559861Z Running tests... 2022-08-17T13:24:12.2560455Z ---------------------------------------------------------------------- 2022-08-17T13:24:12.2578589Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:24:12.2579055Z 2022-08-17T13:24:12.2579516Z ---------------------------------------------------------------------- 2022-08-17T13:24:12.2579991Z Ran 1 test in 0.002s 2022-08-17T13:24:12.2580171Z 2022-08-17T13:24:12.2580247Z OK (skipped=1) 2022-08-17T13:24:12.2580359Z 2022-08-17T13:24:12.2580447Z Generating XML reports... 2022-08-17T13:24:12.2604579Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132412.xml 2022-08-17T13:24:13.4433497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:13.4433890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:13.4435748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:13.4436285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:13.7915017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:13.7924750Z 2022-08-17T13:24:13.7924925Z Running tests... 2022-08-17T13:24:13.7926026Z ---------------------------------------------------------------------- 2022-08-17T13:24:13.7944283Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:24:13.7944760Z 2022-08-17T13:24:13.7945186Z ---------------------------------------------------------------------- 2022-08-17T13:24:13.7945664Z Ran 1 test in 0.002s 2022-08-17T13:24:13.7945873Z 2022-08-17T13:24:13.7945964Z OK (skipped=1) 2022-08-17T13:24:13.7946066Z 2022-08-17T13:24:13.7946154Z Generating XML reports... 2022-08-17T13:24:13.7971592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132413.xml 2022-08-17T13:24:14.9684947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:14.9685574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:14.9709674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:14.9710075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:15.3150083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:15.3160036Z 2022-08-17T13:24:15.3160171Z Running tests... 2022-08-17T13:24:15.3160639Z ---------------------------------------------------------------------- 2022-08-17T13:24:15.3179317Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:24:15.3179819Z 2022-08-17T13:24:15.3180276Z ---------------------------------------------------------------------- 2022-08-17T13:24:15.3180708Z Ran 1 test in 0.002s 2022-08-17T13:24:15.3180830Z 2022-08-17T13:24:15.3180908Z OK (skipped=1) 2022-08-17T13:24:15.3181020Z 2022-08-17T13:24:15.3181106Z Generating XML reports... 2022-08-17T13:24:15.3206141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132415.xml 2022-08-17T13:24:16.4757140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:16.4757533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:16.4760110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:16.4760494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:16.8124218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:16.8133880Z 2022-08-17T13:24:16.8133982Z Running tests... 2022-08-17T13:24:16.8135181Z ---------------------------------------------------------------------- 2022-08-17T13:24:16.8151810Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:24:16.8152252Z 2022-08-17T13:24:16.8152707Z ---------------------------------------------------------------------- 2022-08-17T13:24:16.8152973Z Ran 1 test in 0.002s 2022-08-17T13:24:16.8153079Z 2022-08-17T13:24:16.8153159Z OK (skipped=1) 2022-08-17T13:24:16.8153275Z 2022-08-17T13:24:16.8153368Z Generating XML reports... 2022-08-17T13:24:16.8177710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132416.xml 2022-08-17T13:24:17.9598258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:17.9598700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:17.9601191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:17.9601705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:18.2985539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:18.2995384Z 2022-08-17T13:24:18.2995532Z Running tests... 2022-08-17T13:24:18.2996342Z ---------------------------------------------------------------------- 2022-08-17T13:24:18.3013944Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:24:18.3014559Z 2022-08-17T13:24:18.3014873Z ---------------------------------------------------------------------- 2022-08-17T13:24:18.3015139Z Ran 1 test in 0.002s 2022-08-17T13:24:18.3015262Z 2022-08-17T13:24:18.3015551Z OK (skipped=1) 2022-08-17T13:24:18.3015668Z 2022-08-17T13:24:18.3015760Z Generating XML reports... 2022-08-17T13:24:18.3039258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132418.xml 2022-08-17T13:24:19.4552032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:19.4552410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:19.4555073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:19.4555519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:19.7954333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:19.7965482Z 2022-08-17T13:24:19.7966132Z Running tests... 2022-08-17T13:24:19.7966805Z ---------------------------------------------------------------------- 2022-08-17T13:24:19.7984156Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:24:19.7984568Z 2022-08-17T13:24:19.7984973Z ---------------------------------------------------------------------- 2022-08-17T13:24:19.7985403Z Ran 1 test in 0.002s 2022-08-17T13:24:19.7985885Z 2022-08-17T13:24:19.7986024Z OK (skipped=1) 2022-08-17T13:24:19.7986203Z 2022-08-17T13:24:19.7986362Z Generating XML reports... 2022-08-17T13:24:19.8011569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132419.xml 2022-08-17T13:24:20.9477034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:20.9477681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:20.9480310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:20.9480917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:21.2890095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:21.2900021Z 2022-08-17T13:24:21.2900313Z Running tests... 2022-08-17T13:24:21.2900822Z ---------------------------------------------------------------------- 2022-08-17T13:24:21.2918812Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:21.2919175Z 2022-08-17T13:24:21.2919636Z ---------------------------------------------------------------------- 2022-08-17T13:24:21.2920043Z Ran 1 test in 0.002s 2022-08-17T13:24:21.2920164Z 2022-08-17T13:24:21.2920240Z OK (skipped=1) 2022-08-17T13:24:21.2920356Z 2022-08-17T13:24:21.2920432Z Generating XML reports... 2022-08-17T13:24:21.2945029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132421.xml 2022-08-17T13:24:22.4524664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:22.4525070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:22.4527760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:22.4528142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:22.7959505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:22.7969443Z 2022-08-17T13:24:22.7969914Z Running tests... 2022-08-17T13:24:22.7970343Z ---------------------------------------------------------------------- 2022-08-17T13:24:22.7987421Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:22.7988008Z 2022-08-17T13:24:22.7988271Z ---------------------------------------------------------------------- 2022-08-17T13:24:22.7988515Z Ran 1 test in 0.002s 2022-08-17T13:24:22.7988637Z 2022-08-17T13:24:22.7988721Z OK (skipped=1) 2022-08-17T13:24:22.7988835Z 2022-08-17T13:24:22.7988926Z Generating XML reports... 2022-08-17T13:24:22.8013839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132422.xml 2022-08-17T13:24:23.9459234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:23.9459666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:23.9462219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:23.9462742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:24.2877185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:24.2886649Z 2022-08-17T13:24:24.2886800Z Running tests... 2022-08-17T13:24:24.2887543Z ---------------------------------------------------------------------- 2022-08-17T13:24:24.2905066Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:24.2905378Z 2022-08-17T13:24:24.2905666Z ---------------------------------------------------------------------- 2022-08-17T13:24:24.2905929Z Ran 1 test in 0.002s 2022-08-17T13:24:24.2906051Z 2022-08-17T13:24:24.2906129Z OK (skipped=1) 2022-08-17T13:24:24.2906230Z 2022-08-17T13:24:24.2906321Z Generating XML reports... 2022-08-17T13:24:24.2930139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132424.xml 2022-08-17T13:24:25.4314986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:25.4315390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:25.4317810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:25.4318194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:25.7691910Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:25.7701855Z 2022-08-17T13:24:25.7701946Z Running tests... 2022-08-17T13:24:25.7703231Z ---------------------------------------------------------------------- 2022-08-17T13:24:25.7721247Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:25.7721893Z 2022-08-17T13:24:25.7722330Z ---------------------------------------------------------------------- 2022-08-17T13:24:25.7722594Z Ran 1 test in 0.002s 2022-08-17T13:24:25.7722722Z 2022-08-17T13:24:25.7722799Z OK (skipped=1) 2022-08-17T13:24:25.7722915Z 2022-08-17T13:24:25.7722991Z Generating XML reports... 2022-08-17T13:24:25.7747551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132425.xml 2022-08-17T13:24:26.9049098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:26.9049520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:26.9052135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:26.9052546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:27.2458252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:27.2467693Z 2022-08-17T13:24:27.2467794Z Running tests... 2022-08-17T13:24:27.2468700Z ---------------------------------------------------------------------- 2022-08-17T13:24:27.2487188Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:27.2487514Z 2022-08-17T13:24:27.2487910Z ---------------------------------------------------------------------- 2022-08-17T13:24:27.2488414Z Ran 1 test in 0.002s 2022-08-17T13:24:27.2488612Z 2022-08-17T13:24:27.2488718Z OK (skipped=1) 2022-08-17T13:24:27.2488832Z 2022-08-17T13:24:27.2488922Z Generating XML reports... 2022-08-17T13:24:27.2513860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132427.xml 2022-08-17T13:24:28.3835542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:28.3835915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:28.3838614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:28.3839015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:28.7185847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:28.7195361Z 2022-08-17T13:24:28.7195675Z Running tests... 2022-08-17T13:24:28.7196157Z ---------------------------------------------------------------------- 2022-08-17T13:24:28.7212744Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:28.7213144Z 2022-08-17T13:24:28.7213492Z ---------------------------------------------------------------------- 2022-08-17T13:24:28.7213833Z Ran 1 test in 0.002s 2022-08-17T13:24:28.7213954Z 2022-08-17T13:24:28.7214020Z OK (skipped=1) 2022-08-17T13:24:28.7214134Z 2022-08-17T13:24:28.7214223Z Generating XML reports... 2022-08-17T13:24:28.7238484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132428.xml 2022-08-17T13:24:29.8562675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:29.8563108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:29.8565455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:29.8565836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:30.1983492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:30.1992879Z 2022-08-17T13:24:30.1993018Z Running tests... 2022-08-17T13:24:30.1993412Z ---------------------------------------------------------------------- 2022-08-17T13:24:30.2011461Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:30.2011891Z 2022-08-17T13:24:30.2012244Z ---------------------------------------------------------------------- 2022-08-17T13:24:30.2012504Z Ran 1 test in 0.002s 2022-08-17T13:24:30.2012630Z 2022-08-17T13:24:30.2012707Z OK (skipped=1) 2022-08-17T13:24:30.2012822Z 2022-08-17T13:24:30.2012924Z Generating XML reports... 2022-08-17T13:24:30.2038408Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132430.xml 2022-08-17T13:24:31.3396705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:31.3397336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:31.3398892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:31.3399500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:31.6770965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:31.6781193Z 2022-08-17T13:24:31.6781544Z Running tests... 2022-08-17T13:24:31.6782186Z ---------------------------------------------------------------------- 2022-08-17T13:24:31.6799130Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:31.6799587Z 2022-08-17T13:24:31.6799998Z ---------------------------------------------------------------------- 2022-08-17T13:24:31.6800429Z Ran 1 test in 0.002s 2022-08-17T13:24:31.6800620Z 2022-08-17T13:24:31.6800754Z OK (skipped=1) 2022-08-17T13:24:31.6800955Z 2022-08-17T13:24:31.6801117Z Generating XML reports... 2022-08-17T13:24:31.6826263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132431.xml 2022-08-17T13:24:32.8185848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:32.8186589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:32.8189269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:32.8190177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:33.1685355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:33.1695184Z 2022-08-17T13:24:33.1695500Z Running tests... 2022-08-17T13:24:33.1696152Z ---------------------------------------------------------------------- 2022-08-17T13:24:33.1714689Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:33.1715337Z 2022-08-17T13:24:33.1715773Z ---------------------------------------------------------------------- 2022-08-17T13:24:33.1716303Z Ran 1 test in 0.002s 2022-08-17T13:24:33.1716500Z 2022-08-17T13:24:33.1716579Z OK (skipped=1) 2022-08-17T13:24:33.1716678Z 2022-08-17T13:24:33.1716768Z Generating XML reports... 2022-08-17T13:24:33.1740919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132433.xml 2022-08-17T13:24:34.3104040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:34.3104467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:34.3107098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:34.3107509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:34.6515840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:34.6524864Z 2022-08-17T13:24:34.6525031Z Running tests... 2022-08-17T13:24:34.6525635Z ---------------------------------------------------------------------- 2022-08-17T13:24:34.6543209Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:34.6543690Z 2022-08-17T13:24:34.6544013Z ---------------------------------------------------------------------- 2022-08-17T13:24:34.6544279Z Ran 1 test in 0.002s 2022-08-17T13:24:34.6544400Z 2022-08-17T13:24:34.6544463Z OK (skipped=1) 2022-08-17T13:24:34.6544577Z 2022-08-17T13:24:34.6544666Z Generating XML reports... 2022-08-17T13:24:34.6570144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132434.xml 2022-08-17T13:24:35.7948293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:35.7948677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:35.7951634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:35.7952023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:36.1330229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:36.1340301Z 2022-08-17T13:24:36.1340449Z Running tests... 2022-08-17T13:24:36.1341061Z ---------------------------------------------------------------------- 2022-08-17T13:24:36.1357926Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:36.1358338Z 2022-08-17T13:24:36.1358599Z ---------------------------------------------------------------------- 2022-08-17T13:24:36.1358917Z Ran 1 test in 0.002s 2022-08-17T13:24:36.1359042Z 2022-08-17T13:24:36.1359124Z OK (skipped=1) 2022-08-17T13:24:36.1359239Z 2022-08-17T13:24:36.1359326Z Generating XML reports... 2022-08-17T13:24:36.1383828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132436.xml 2022-08-17T13:24:37.3018274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:37.3018966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:37.3021234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:37.3021620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:37.6444675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:37.6454841Z 2022-08-17T13:24:37.6454948Z Running tests... 2022-08-17T13:24:37.6455370Z ---------------------------------------------------------------------- 2022-08-17T13:24:37.6473821Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:37.6474333Z 2022-08-17T13:24:37.6474762Z ---------------------------------------------------------------------- 2022-08-17T13:24:37.6475192Z Ran 1 test in 0.002s 2022-08-17T13:24:37.6475390Z 2022-08-17T13:24:37.6475526Z OK (skipped=1) 2022-08-17T13:24:37.6475723Z 2022-08-17T13:24:37.6475864Z Generating XML reports... 2022-08-17T13:24:37.6500909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132437.xml 2022-08-17T13:24:38.8109433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:38.8110087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:38.8112748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:38.8113346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:39.1598280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:39.1608275Z 2022-08-17T13:24:39.1608421Z Running tests... 2022-08-17T13:24:39.1609080Z ---------------------------------------------------------------------- 2022-08-17T13:24:39.1627145Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:39.1627615Z 2022-08-17T13:24:39.1628005Z ---------------------------------------------------------------------- 2022-08-17T13:24:39.1628439Z Ran 1 test in 0.002s 2022-08-17T13:24:39.1628575Z 2022-08-17T13:24:39.1628650Z OK (skipped=1) 2022-08-17T13:24:39.1628764Z 2022-08-17T13:24:39.1628855Z Generating XML reports... 2022-08-17T13:24:39.1653660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132439.xml 2022-08-17T13:24:40.3288926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:40.3289313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:40.3291501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:40.3291860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:40.6740082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:40.6750056Z 2022-08-17T13:24:40.6750172Z Running tests... 2022-08-17T13:24:40.6750617Z ---------------------------------------------------------------------- 2022-08-17T13:24:40.6768442Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:40.6768811Z 2022-08-17T13:24:40.6769103Z ---------------------------------------------------------------------- 2022-08-17T13:24:40.6769365Z Ran 1 test in 0.002s 2022-08-17T13:24:40.6769485Z 2022-08-17T13:24:40.6769547Z OK (skipped=1) 2022-08-17T13:24:40.6769658Z 2022-08-17T13:24:40.6769803Z Generating XML reports... 2022-08-17T13:24:40.6794298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132440.xml 2022-08-17T13:24:41.8444897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:41.8445285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:41.8447695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:41.8448070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:42.1940557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:42.1950449Z 2022-08-17T13:24:42.1950574Z Running tests... 2022-08-17T13:24:42.1951155Z ---------------------------------------------------------------------- 2022-08-17T13:24:42.1970920Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:42.1971303Z 2022-08-17T13:24:42.1971591Z ---------------------------------------------------------------------- 2022-08-17T13:24:42.1971925Z Ran 1 test in 0.002s 2022-08-17T13:24:42.1972047Z 2022-08-17T13:24:42.1972128Z OK (skipped=1) 2022-08-17T13:24:42.1972229Z 2022-08-17T13:24:42.1972319Z Generating XML reports... 2022-08-17T13:24:42.1997113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132442.xml 2022-08-17T13:24:43.3773465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:43.3774170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:43.3777423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:43.3778049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:43.7219689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:43.7230486Z 2022-08-17T13:24:43.7230815Z Running tests... 2022-08-17T13:24:43.7231478Z ---------------------------------------------------------------------- 2022-08-17T13:24:43.7248901Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:24:43.7249365Z 2022-08-17T13:24:43.7249809Z ---------------------------------------------------------------------- 2022-08-17T13:24:43.7250100Z Ran 1 test in 0.002s 2022-08-17T13:24:43.7250481Z 2022-08-17T13:24:43.7250559Z OK (skipped=1) 2022-08-17T13:24:43.7250675Z 2022-08-17T13:24:43.7250749Z Generating XML reports... 2022-08-17T13:24:43.7274509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132443.xml 2022-08-17T13:24:44.8758879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:44.8759292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:44.8761929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:44.8762308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:45.2134167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:45.2144052Z 2022-08-17T13:24:45.2144215Z Running tests... 2022-08-17T13:24:45.2144728Z ---------------------------------------------------------------------- 2022-08-17T13:24:45.2163874Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:24:45.2164270Z 2022-08-17T13:24:45.2164609Z ---------------------------------------------------------------------- 2022-08-17T13:24:45.2165344Z Ran 1 test in 0.002s 2022-08-17T13:24:45.2165517Z 2022-08-17T13:24:45.2165582Z OK (skipped=1) 2022-08-17T13:24:45.2165699Z 2022-08-17T13:24:45.2165790Z Generating XML reports... 2022-08-17T13:24:45.2190124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132445.xml 2022-08-17T13:24:46.3551160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:46.3551550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:46.3554355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:46.3554733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:46.6974284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:46.6984098Z 2022-08-17T13:24:46.6984211Z Running tests... 2022-08-17T13:24:46.6984772Z ---------------------------------------------------------------------- 2022-08-17T13:24:46.9626357Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10928 2022-08-17T13:24:46.9655714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10929 2022-08-17T13:24:46.9684024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10930 2022-08-17T13:24:47.9538414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:47.9538836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:47.9541771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:47.9542256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:47.9723368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:47.9723736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:47.9727597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:47.9728240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:47.9742581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:47.9743278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:47.9747016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:47.9747634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:48.3413894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:24:48.3514392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:24:48.3515008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:24:48.3515770Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:48.3516573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:48.3517474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:48.3622031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:24:48.4528679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:24:48.4529745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:24:48.7732923Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T13:24:48.7733144Z 2022-08-17T13:24:48.7733459Z ---------------------------------------------------------------------- 2022-08-17T13:24:48.7733719Z Ran 1 test in 2.075s 2022-08-17T13:24:48.7733825Z 2022-08-17T13:24:48.7733903Z OK (skipped=1) 2022-08-17T13:24:48.7734013Z 2022-08-17T13:24:48.7734102Z Generating XML reports... 2022-08-17T13:24:48.7769680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132446.xml 2022-08-17T13:24:50.0912981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:50.0913360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:50.0915839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:50.0916213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:50.4327943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:50.4337700Z 2022-08-17T13:24:50.4337799Z Running tests... 2022-08-17T13:24:50.4338733Z ---------------------------------------------------------------------- 2022-08-17T13:24:50.4356584Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:24:50.4357051Z 2022-08-17T13:24:50.4357413Z ---------------------------------------------------------------------- 2022-08-17T13:24:50.4357857Z Ran 1 test in 0.002s 2022-08-17T13:24:50.4358032Z 2022-08-17T13:24:50.4358156Z OK (skipped=1) 2022-08-17T13:24:50.4358354Z 2022-08-17T13:24:50.4358510Z Generating XML reports... 2022-08-17T13:24:50.4384096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132450.xml 2022-08-17T13:24:51.5834183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:51.5834591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:51.5837322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:51.5837703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:51.9226560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:51.9235779Z 2022-08-17T13:24:51.9235891Z Running tests... 2022-08-17T13:24:51.9236337Z ---------------------------------------------------------------------- 2022-08-17T13:24:51.9254867Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:24:51.9255294Z 2022-08-17T13:24:51.9255695Z ---------------------------------------------------------------------- 2022-08-17T13:24:51.9256134Z Ran 1 test in 0.002s 2022-08-17T13:24:51.9256341Z 2022-08-17T13:24:51.9256476Z OK (skipped=1) 2022-08-17T13:24:51.9256678Z 2022-08-17T13:24:51.9256826Z Generating XML reports... 2022-08-17T13:24:51.9281705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132451.xml 2022-08-17T13:24:53.0587737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:53.0588378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:53.0591025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:53.0591608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:53.4037709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:53.4048054Z 2022-08-17T13:24:53.4048372Z Running tests... 2022-08-17T13:24:53.4049012Z ---------------------------------------------------------------------- 2022-08-17T13:24:53.6733433Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11001 2022-08-17T13:24:53.6762549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11002 2022-08-17T13:24:53.6792698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11003 2022-08-17T13:24:54.6390662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:54.6391334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:54.6393916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:54.6394554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:54.6395514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:54.6396105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:54.6400809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:54.6401490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:54.6790834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:54.6791573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:54.6794225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:54.6794938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:55.0502936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:24:55.0503706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:24:55.0504325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:24:55.0505170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:55.0505740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:55.0506923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:24:55.0512237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:24:55.0512667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:24:55.1513956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:24:56.7861674Z ok (3.381s) 2022-08-17T13:24:56.7861937Z 2022-08-17T13:24:56.7862432Z ---------------------------------------------------------------------- 2022-08-17T13:24:56.7862756Z Ran 1 test in 3.381s 2022-08-17T13:24:56.7862878Z 2022-08-17T13:24:56.7862927Z OK 2022-08-17T13:24:56.7863026Z 2022-08-17T13:24:56.7863228Z Generating XML reports... 2022-08-17T13:24:56.7897654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132453.xml 2022-08-17T13:24:58.0752044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:58.0752730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:58.0755998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:58.0756604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:58.4199394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:24:58.4209724Z 2022-08-17T13:24:58.4209898Z Running tests... 2022-08-17T13:24:58.4210410Z ---------------------------------------------------------------------- 2022-08-17T13:24:58.6856602Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11054 2022-08-17T13:24:58.6884433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11055 2022-08-17T13:24:58.6913256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11056 2022-08-17T13:24:59.6639356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:59.6640043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:59.6640854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:59.6641422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:59.6642213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:59.6642808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:59.6644117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:59.6644538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:24:59.6677653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:24:59.6678093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:24:59.6681723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:24:59.6682203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:00.0312995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:00.0414398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:00.0414981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:00.0415821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:00.0416374Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:00.0416934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:00.0422998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:00.0423661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:00.0424445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:00.3959269Z skip: CUDA is not available. (1.975s) 2022-08-17T13:25:00.3959599Z 2022-08-17T13:25:00.3960035Z ---------------------------------------------------------------------- 2022-08-17T13:25:00.3960313Z Ran 1 test in 1.975s 2022-08-17T13:25:00.3960435Z 2022-08-17T13:25:00.3960511Z OK (skipped=1) 2022-08-17T13:25:00.3960622Z 2022-08-17T13:25:00.3960710Z Generating XML reports... 2022-08-17T13:25:00.3998606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132458.xml 2022-08-17T13:25:01.7025931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:01.7026613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:01.7028968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:01.7029571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:02.0510201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:02.0519918Z 2022-08-17T13:25:02.0520381Z Running tests... 2022-08-17T13:25:02.0520792Z ---------------------------------------------------------------------- 2022-08-17T13:25:02.3174345Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11107 2022-08-17T13:25:02.3202122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11108 2022-08-17T13:25:02.3231474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11109 2022-08-17T13:25:03.3584143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:03.3584770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:03.3587978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:03.3588628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:03.3654444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:03.3655039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:03.3659385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:03.3771665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:03.3772676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:03.3775911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:03.3776795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:03.3777438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:03.7425458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:03.7527192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:03.7527706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:03.7528638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:03.7529572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:03.7530147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:03.7535905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:03.7536597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:03.7536983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:03.7743926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:25:03.7843986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:25:03.7844539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:25:03.7845333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:03.7846077Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:03.7846625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:05.4297967Z ok (3.377s) 2022-08-17T13:25:05.4298236Z 2022-08-17T13:25:05.4298785Z ---------------------------------------------------------------------- 2022-08-17T13:25:05.4299067Z Ran 1 test in 3.378s 2022-08-17T13:25:05.4299191Z 2022-08-17T13:25:05.4299256Z OK 2022-08-17T13:25:05.4299352Z 2022-08-17T13:25:05.4299451Z Generating XML reports... 2022-08-17T13:25:05.4334103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132502.xml 2022-08-17T13:25:06.7395601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:06.7395979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:06.7398361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:06.7398774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:07.0813145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:07.0822798Z 2022-08-17T13:25:07.0822877Z Running tests... 2022-08-17T13:25:07.0823536Z ---------------------------------------------------------------------- 2022-08-17T13:25:07.3496530Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11169 2022-08-17T13:25:07.3525223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11170 2022-08-17T13:25:07.3553190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11171 2022-08-17T13:25:08.3636888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:08.3637582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:08.3641129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:08.3642114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:08.3758115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:08.3758683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:08.3761967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:08.3762611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:08.3914574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:08.3915252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:08.3918400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:08.3918827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:08.7602139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:08.7602757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:08.7603321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:08.7603970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:08.7604512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:08.7605061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:08.7709669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:08.8615252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:08.8615882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:09.1600550Z skip: CUDA is not available. (2.077s) 2022-08-17T13:25:09.1600845Z 2022-08-17T13:25:09.1601205Z ---------------------------------------------------------------------- 2022-08-17T13:25:09.1601474Z Ran 1 test in 2.078s 2022-08-17T13:25:09.1601599Z 2022-08-17T13:25:09.1601674Z OK (skipped=1) 2022-08-17T13:25:09.1601774Z 2022-08-17T13:25:09.1601866Z Generating XML reports... 2022-08-17T13:25:09.1639939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132507.xml 2022-08-17T13:25:10.4680667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:10.4681050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:10.4684120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:10.4684504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:10.8169444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:10.8179703Z 2022-08-17T13:25:10.8179848Z Running tests... 2022-08-17T13:25:10.8180647Z ---------------------------------------------------------------------- 2022-08-17T13:25:11.0825668Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11222 2022-08-17T13:25:11.0855108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11223 2022-08-17T13:25:11.0883774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11224 2022-08-17T13:25:12.0397033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:12.0398041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:12.0399814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:12.0400431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:12.0871669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:12.0872339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:12.0876033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:12.0876722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:12.1048104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:12.1048677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:12.1051295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:12.1051725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:12.4747256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:12.4847390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:12.4848003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:12.4848968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:12.4849895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:12.4850809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:12.4955430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:12.4956050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:12.5162150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:25:12.5162571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:25:12.5858633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:12.5861381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:25:12.5862028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:12.5869811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:12.5870620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:13.7944309Z ok (2.976s) 2022-08-17T13:25:13.7944563Z 2022-08-17T13:25:13.7945090Z ---------------------------------------------------------------------- 2022-08-17T13:25:13.7945507Z Ran 1 test in 2.976s 2022-08-17T13:25:13.7945631Z 2022-08-17T13:25:13.7945695Z OK 2022-08-17T13:25:13.7945778Z 2022-08-17T13:25:13.7945874Z Generating XML reports... 2022-08-17T13:25:13.7980410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132510.xml 2022-08-17T13:25:15.0794278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:15.0794908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:15.0797204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:15.0797740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:15.4264637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:15.4273864Z 2022-08-17T13:25:15.4274010Z Running tests... 2022-08-17T13:25:15.4274637Z ---------------------------------------------------------------------- 2022-08-17T13:25:15.6898040Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11281 2022-08-17T13:25:15.6926444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11282 2022-08-17T13:25:15.6954259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11283 2022-08-17T13:25:16.6920062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:16.6920564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:16.6922814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:16.6923218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:16.7036630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:16.7037207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:16.7040975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:16.7041395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:16.7116138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:16.7116516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:16.7120654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:16.7121342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:17.0729326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:17.0830481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:17.0831175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:17.0832121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:17.0832692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:17.0833575Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:17.0839034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:17.0839695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:17.0840268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:17.4001497Z skip: CUDA is not available. (1.972s) 2022-08-17T13:25:17.4001819Z 2022-08-17T13:25:17.4002164Z ---------------------------------------------------------------------- 2022-08-17T13:25:17.4002449Z Ran 1 test in 1.973s 2022-08-17T13:25:17.4002556Z 2022-08-17T13:25:17.4002635Z OK (skipped=1) 2022-08-17T13:25:17.4002751Z 2022-08-17T13:25:17.4002843Z Generating XML reports... 2022-08-17T13:25:17.4037196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132515.xml 2022-08-17T13:25:18.6626061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:18.6626547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:18.6628911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:18.6629292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:19.0054421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:19.0064328Z 2022-08-17T13:25:19.0064478Z Running tests... 2022-08-17T13:25:19.0064918Z ---------------------------------------------------------------------- 2022-08-17T13:25:19.2724444Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11334 2022-08-17T13:25:19.2753404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11335 2022-08-17T13:25:19.2782569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11336 2022-08-17T13:25:20.2754305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:20.2754954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:20.2758014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:20.2758692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:20.2840204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:20.2840613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:20.2844104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:20.2844583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:20.2977986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:20.2978398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:20.2981553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:20.2981928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:20.6663206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:20.6764867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:20.6765330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:20.6766407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:20.6767362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:20.6767955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:20.6774810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:20.6775261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:20.6775708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:20.6980937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:25:20.7083152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:25:20.7083868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:25:20.7084828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:20.7085611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:20.7086420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:22.1846665Z ok (3.178s) 2022-08-17T13:25:22.1846919Z 2022-08-17T13:25:22.1847472Z ---------------------------------------------------------------------- 2022-08-17T13:25:22.1847812Z Ran 1 test in 3.178s 2022-08-17T13:25:22.1847921Z 2022-08-17T13:25:22.1848003Z OK 2022-08-17T13:25:22.1848100Z 2022-08-17T13:25:22.1848196Z Generating XML reports... 2022-08-17T13:25:22.1882350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132519.xml 2022-08-17T13:25:23.4816745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:23.4817157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:23.4819075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:23.4819436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:23.8232140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:23.8241843Z 2022-08-17T13:25:23.8241938Z Running tests... 2022-08-17T13:25:23.8242413Z ---------------------------------------------------------------------- 2022-08-17T13:25:23.8263363Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s) 2022-08-17T13:25:23.8263798Z 2022-08-17T13:25:23.8264166Z ---------------------------------------------------------------------- 2022-08-17T13:25:23.8264513Z Ran 1 test in 0.002s 2022-08-17T13:25:23.8264662Z 2022-08-17T13:25:23.8264778Z OK (skipped=1) 2022-08-17T13:25:23.8264973Z 2022-08-17T13:25:23.8265141Z Generating XML reports... 2022-08-17T13:25:23.8289716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132523.xml 2022-08-17T13:25:24.9907863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:24.9908245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:24.9910190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:24.9910568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:25.3324141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:25.3334265Z 2022-08-17T13:25:25.3334401Z Running tests... 2022-08-17T13:25:25.3335018Z ---------------------------------------------------------------------- 2022-08-17T13:25:25.5963282Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11406 2022-08-17T13:25:25.5991164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11407 2022-08-17T13:25:25.6020055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11408 2022-08-17T13:25:26.5689082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:26.5689806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:26.5692172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:26.5692658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:26.5699507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:26.5700031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:26.5704649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:26.5705287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:26.6246876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:26.6247451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:26.6249652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:26.6250169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:26.9852909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:26.9954158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:26.9955223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:26.9955640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:26.9956168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:26.9956727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:26.9962999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:26.9964144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:26.9964978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:25:26.9965439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:27.0072596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:25:27.0073061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:25:27.0073713Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:27.0074303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:27.0168833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:25:32.3139682Z ok (6.980s) 2022-08-17T13:25:32.3139947Z 2022-08-17T13:25:32.3140476Z ---------------------------------------------------------------------- 2022-08-17T13:25:32.3140820Z Ran 1 test in 6.980s 2022-08-17T13:25:32.3140944Z 2022-08-17T13:25:32.3141010Z OK 2022-08-17T13:25:32.3141107Z 2022-08-17T13:25:32.3141207Z Generating XML reports... 2022-08-17T13:25:32.3177222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132525.xml 2022-08-17T13:25:33.5919818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:33.5920759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:33.5923005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:33.5923602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:33.9303277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:33.9313890Z 2022-08-17T13:25:33.9314328Z Running tests... 2022-08-17T13:25:33.9314732Z ---------------------------------------------------------------------- 2022-08-17T13:25:34.1964166Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11465 2022-08-17T13:25:34.1991163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11466 2022-08-17T13:25:34.2019761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11467 2022-08-17T13:25:35.1942213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:35.1942952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:35.1946213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:35.1946905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:35.2179876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:35.2180554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:35.2184265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:35.2184655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:35.2598756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:35.2599481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:35.2603157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:35.2603797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:35.6189636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:35.6290864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:35.6291442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:35.6292083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:35.6292639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:35.6293205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:35.6298437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:35.6298918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:35.6300315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:36.0067826Z ok (2.075s) 2022-08-17T13:25:36.0068213Z 2022-08-17T13:25:36.0068701Z ---------------------------------------------------------------------- 2022-08-17T13:25:36.0069124Z Ran 1 test in 2.075s 2022-08-17T13:25:36.0069314Z 2022-08-17T13:25:36.0069419Z OK 2022-08-17T13:25:36.0069550Z 2022-08-17T13:25:36.0069705Z Generating XML reports... 2022-08-17T13:25:36.0104981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132533.xml 2022-08-17T13:25:37.2853966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:37.2854435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:37.2857147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:37.2857582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:37.6210955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:37.6220947Z 2022-08-17T13:25:37.6221205Z Running tests... 2022-08-17T13:25:37.6221643Z ---------------------------------------------------------------------- 2022-08-17T13:25:37.8817367Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11518 2022-08-17T13:25:37.8845341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11519 2022-08-17T13:25:37.8873939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11520 2022-08-17T13:25:38.8765646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:38.8766563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:38.8768773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:38.8769349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:38.8798535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:38.8798928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:38.8802511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:38.8802952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:38.8857682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:38.8858118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:38.8861943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:38.8862438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:39.2433401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:39.2533656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:39.2534382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:39.2535034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:39.2535597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:39.2536135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:39.2641953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:39.3546779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:39.3547250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:39.7926749Z ok (2.170s) 2022-08-17T13:25:39.7927016Z 2022-08-17T13:25:39.7927495Z ---------------------------------------------------------------------- 2022-08-17T13:25:39.7928267Z Ran 1 test in 2.170s 2022-08-17T13:25:39.7928482Z 2022-08-17T13:25:39.7928597Z OK 2022-08-17T13:25:39.7928769Z 2022-08-17T13:25:39.7928939Z Generating XML reports... 2022-08-17T13:25:39.7965513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132537.xml 2022-08-17T13:25:41.0798207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:41.0798588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:41.0800475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:41.0800985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:41.4241524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:41.4244122Z 2022-08-17T13:25:41.4244291Z Running tests... 2022-08-17T13:25:41.4244976Z ---------------------------------------------------------------------- 2022-08-17T13:25:41.4267597Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:41.4268090Z 2022-08-17T13:25:41.4268638Z ---------------------------------------------------------------------- 2022-08-17T13:25:41.4268983Z Ran 1 test in 0.002s 2022-08-17T13:25:41.4269104Z 2022-08-17T13:25:41.4269183Z OK (skipped=1) 2022-08-17T13:25:41.4269299Z 2022-08-17T13:25:41.4269374Z Generating XML reports... 2022-08-17T13:25:41.4294113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132541.xml 2022-08-17T13:25:42.5888813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:42.5889205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:42.5892082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:42.5892463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:42.9320888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:42.9331487Z 2022-08-17T13:25:42.9331769Z Running tests... 2022-08-17T13:25:42.9332419Z ---------------------------------------------------------------------- 2022-08-17T13:25:42.9359924Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2022-08-17T13:25:42.9360290Z 2022-08-17T13:25:42.9360612Z ---------------------------------------------------------------------- 2022-08-17T13:25:42.9360899Z Ran 1 test in 0.003s 2022-08-17T13:25:42.9361027Z 2022-08-17T13:25:42.9361104Z OK (skipped=1) 2022-08-17T13:25:42.9361219Z 2022-08-17T13:25:42.9361308Z Generating XML reports... 2022-08-17T13:25:42.9386036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132542.xml 2022-08-17T13:25:44.0880461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:44.0880906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:44.0883222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:44.0883600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:44.4278434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:44.4288347Z 2022-08-17T13:25:44.4288476Z Running tests... 2022-08-17T13:25:44.4289098Z ---------------------------------------------------------------------- 2022-08-17T13:25:44.4315548Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2022-08-17T13:25:44.4316253Z 2022-08-17T13:25:44.4316644Z ---------------------------------------------------------------------- 2022-08-17T13:25:44.4317106Z Ran 1 test in 0.003s 2022-08-17T13:25:44.4317317Z 2022-08-17T13:25:44.4317453Z OK (skipped=1) 2022-08-17T13:25:44.4317663Z 2022-08-17T13:25:44.4317806Z Generating XML reports... 2022-08-17T13:25:44.4343413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132544.xml 2022-08-17T13:25:45.5756993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:45.5757394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:45.5759470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:45.5760039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:45.9153546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:45.9163341Z 2022-08-17T13:25:45.9163460Z Running tests... 2022-08-17T13:25:45.9164103Z ---------------------------------------------------------------------- 2022-08-17T13:25:45.9187312Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:45.9187785Z 2022-08-17T13:25:45.9188069Z ---------------------------------------------------------------------- 2022-08-17T13:25:45.9188318Z Ran 1 test in 0.002s 2022-08-17T13:25:45.9188438Z 2022-08-17T13:25:45.9188515Z OK (skipped=1) 2022-08-17T13:25:45.9188634Z 2022-08-17T13:25:45.9188723Z Generating XML reports... 2022-08-17T13:25:45.9212503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132545.xml 2022-08-17T13:25:47.0626361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:47.0626925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:47.0628102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:47.0628663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:47.4052844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:47.4062132Z 2022-08-17T13:25:47.4062273Z Running tests... 2022-08-17T13:25:47.4062972Z ---------------------------------------------------------------------- 2022-08-17T13:25:47.4081887Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:47.4082337Z 2022-08-17T13:25:47.4082805Z ---------------------------------------------------------------------- 2022-08-17T13:25:47.4083226Z Ran 1 test in 0.002s 2022-08-17T13:25:47.4083348Z 2022-08-17T13:25:47.4083422Z OK (skipped=1) 2022-08-17T13:25:47.4083522Z 2022-08-17T13:25:47.4083615Z Generating XML reports... 2022-08-17T13:25:47.4109435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132547.xml 2022-08-17T13:25:48.5583765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:48.5584177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:48.5586509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:48.5586881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:48.9059457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:48.9070130Z 2022-08-17T13:25:48.9070626Z Running tests... 2022-08-17T13:25:48.9071284Z ---------------------------------------------------------------------- 2022-08-17T13:25:48.9093971Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:48.9094441Z 2022-08-17T13:25:48.9094788Z ---------------------------------------------------------------------- 2022-08-17T13:25:48.9095331Z Ran 1 test in 0.002s 2022-08-17T13:25:48.9095547Z 2022-08-17T13:25:48.9095668Z OK (skipped=1) 2022-08-17T13:25:48.9095852Z 2022-08-17T13:25:48.9095995Z Generating XML reports... 2022-08-17T13:25:48.9121744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132548.xml 2022-08-17T13:25:50.0866022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:50.0866657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:50.0868865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:50.0869455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:50.4301237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:50.4312253Z 2022-08-17T13:25:50.4312622Z Running tests... 2022-08-17T13:25:50.4313317Z ---------------------------------------------------------------------- 2022-08-17T13:25:50.4336495Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:50.4336930Z 2022-08-17T13:25:50.4337344Z ---------------------------------------------------------------------- 2022-08-17T13:25:50.4337766Z Ran 1 test in 0.002s 2022-08-17T13:25:50.4337976Z 2022-08-17T13:25:50.4338109Z OK (skipped=1) 2022-08-17T13:25:50.4338298Z 2022-08-17T13:25:50.4338465Z Generating XML reports... 2022-08-17T13:25:50.4363409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132550.xml 2022-08-17T13:25:51.5899630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:51.5900044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:51.5902595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:51.5903121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:51.9300101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:51.9310091Z 2022-08-17T13:25:51.9310368Z Running tests... 2022-08-17T13:25:51.9310777Z ---------------------------------------------------------------------- 2022-08-17T13:25:51.9330772Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:25:51.9331238Z 2022-08-17T13:25:51.9331664Z ---------------------------------------------------------------------- 2022-08-17T13:25:51.9332091Z Ran 1 test in 0.002s 2022-08-17T13:25:51.9332285Z 2022-08-17T13:25:51.9332398Z OK (skipped=1) 2022-08-17T13:25:51.9332578Z 2022-08-17T13:25:51.9332749Z Generating XML reports... 2022-08-17T13:25:51.9358110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132551.xml 2022-08-17T13:25:53.0807718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:53.0808133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:53.0810999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:53.0811386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:53.4214664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:53.4224886Z 2022-08-17T13:25:53.4225055Z Running tests... 2022-08-17T13:25:53.4225513Z ---------------------------------------------------------------------- 2022-08-17T13:25:53.6835399Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11651 2022-08-17T13:25:53.6863355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11652 2022-08-17T13:25:53.6891412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11653 2022-08-17T13:25:54.6631156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:54.6631821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:54.6634036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:54.6634697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:54.6690726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:54.6691644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:54.6694838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:54.6695460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:54.6702528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:54.6703257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:54.6707832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:54.6708462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:55.0329750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:25:55.0429569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:25:55.0430220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:25:55.0431170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:55.0432086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:55.0433056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:25:55.0539805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:25:55.0540218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:25:55.0543854Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.0544512Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1441464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:25:55.1445790Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1451661Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1452430Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1453179Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1460172Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1466003Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1466500Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1470812Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1471615Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1472329Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1475299Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1478097Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1481440Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1484813Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1488122Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1488732Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1489204Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1493673Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1494790Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1505845Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1506203Z 2022-08-17T13:25:55.1506565Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1507068Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1507829Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1508545Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1509267Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1510129Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1510665Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1514996Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1515446Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1518715Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1522155Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1523005Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1523826Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1530013Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1530552Z 2022-08-17T13:25:55.1531160Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1533123Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1536837Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1537672Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1541695Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1542188Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1545577Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1548638Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1549132Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1549598Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1553233Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1554969Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1555499Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1559298Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1560060Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1560514Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1563790Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1565508Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1566172Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1569388Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1569987Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1573332Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1573794Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1576604Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1577103Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1579904Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1580728Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1581549Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1584307Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1586165Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1589322Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1592683Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1593006Z 2022-08-17T13:25:55.1593297Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1597371Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1599466Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1599948Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1606882Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1607728Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1608042Z 2022-08-17T13:25:55.1608306Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1612512Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1612872Z 2022-08-17T13:25:55.1615858Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1616339Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1616808Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1621305Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1623083Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1623874Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1626995Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1627643Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1627953Z 2022-08-17T13:25:55.1634033Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1634847Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1635515Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1638036Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1638526Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1639776Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1641859Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1645002Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1646797Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1650159Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1651354Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1652084Z 2022-08-17T13:25:55.1655755Z STAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1656213Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1658162Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:25:55.1661561Z STAGE:2022-08-17 13:25:55 11653:11653 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:25:55 11652:11652 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.1661869Z 2022-08-17T13:25:55.1662149Z STAGE:2022-08-17 13:25:55 11651:11651 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:25:55.5941132Z ok (2.171s) 2022-08-17T13:25:55.5941374Z 2022-08-17T13:25:55.5941855Z ---------------------------------------------------------------------- 2022-08-17T13:25:55.5942268Z Ran 1 test in 2.172s 2022-08-17T13:25:55.5942458Z 2022-08-17T13:25:55.5942556Z OK 2022-08-17T13:25:55.5942681Z 2022-08-17T13:25:55.5942918Z Generating XML reports... 2022-08-17T13:25:55.5979439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132553.xml 2022-08-17T13:25:56.8696673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:56.8697043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:56.8699451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:56.8699830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:57.2073454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:57.2083669Z 2022-08-17T13:25:57.2083766Z Running tests... 2022-08-17T13:25:57.2084770Z ---------------------------------------------------------------------- 2022-08-17T13:25:57.4592564Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.250s) 2022-08-17T13:25:57.4593121Z 2022-08-17T13:25:57.4593351Z ---------------------------------------------------------------------- 2022-08-17T13:25:57.4593596Z Ran 1 test in 0.251s 2022-08-17T13:25:57.4593715Z 2022-08-17T13:25:57.4593791Z OK (skipped=1) 2022-08-17T13:25:57.4593903Z 2022-08-17T13:25:57.4593991Z Generating XML reports... 2022-08-17T13:25:57.4617066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132557.xml 2022-08-17T13:25:58.7013269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:25:58.7013665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:25:58.7016047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:25:58.7016437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:25:59.0474510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:25:59.0484718Z 2022-08-17T13:25:59.0484860Z Running tests... 2022-08-17T13:25:59.0485253Z ---------------------------------------------------------------------- 2022-08-17T13:25:59.3151260Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11717 2022-08-17T13:25:59.3179079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11718 2022-08-17T13:25:59.3207788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11719 2022-08-17T13:26:00.2978564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:00.2979199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:00.2981975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:00.2982384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:00.3068020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:00.3068470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:00.3071611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:00.3072010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:00.3086098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:00.3086691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:00.3091284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:00.3091943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:00.6849266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:00.6949724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:00.6950344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:00.6951385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:00.6952237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:00.6953184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:00.7056990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:00.7963787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:00.7964268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:00.8272979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:26:00.8273643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:26:00.8274308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:26:00.8275270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:00.8276139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:00.8277059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:00.8277865Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8278336Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8278885Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8283053Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8283981Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8284293Z 2022-08-17T13:26:00.8291575Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8301607Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8301994Z 2022-08-17T13:26:00.8306303Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8307150Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8307474Z 2022-08-17T13:26:00.8312853Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8314163Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8317294Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8319754Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8322966Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8324132Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8327938Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8329326Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8336808Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8340288Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8341091Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8341550Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8347274Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8348142Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8349879Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8353103Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8353912Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8358328Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8368726Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8369552Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8370395Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8371114Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8371834Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8372851Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8374140Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8378715Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8379939Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8384076Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8385252Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8385768Z 2022-08-17T13:26:00.8389142Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8389910Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8390702Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8394095Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8394585Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8395039Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8399663Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8401676Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8402530Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8405652Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8406681Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8407483Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8409909Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8413258Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8413961Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8416064Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8417323Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8418188Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8422472Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8423023Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8425968Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8428633Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8429230Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8429776Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8433030Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8435908Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8436786Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8439271Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8439945Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8440442Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8444725Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8445413Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8447168Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8450608Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8450918Z 2022-08-17T13:26:00.8451191Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8455504Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8456754Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8457546Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8467345Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8467928Z 2022-08-17T13:26:00.8468351Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8469090Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8469846Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8470614Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8471407Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8472504Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8473030Z 2022-08-17T13:26:00.8475414Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8479343Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8479769Z 2022-08-17T13:26:00.8483266Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8484089Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8484942Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8488810Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8491755Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8492357Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8504015Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8505927Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8507061Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8508120Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8509150Z STAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8510134Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:00.8511097Z STAGE:2022-08-17 13:26:00 11718:11718 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8512347Z STAGE:2022-08-17 13:26:00 11717:11717 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:00 11719:11719 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:00.8512861Z 2022-08-17T13:26:01.3259200Z ok (2.277s) 2022-08-17T13:26:01.3259765Z 2022-08-17T13:26:01.3260137Z ---------------------------------------------------------------------- 2022-08-17T13:26:01.3260405Z Ran 1 test in 2.277s 2022-08-17T13:26:01.3260530Z 2022-08-17T13:26:01.3260582Z OK 2022-08-17T13:26:01.3260681Z 2022-08-17T13:26:01.3260783Z Generating XML reports... 2022-08-17T13:26:01.3296411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132559.xml 2022-08-17T13:26:02.6436416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:02.6436828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:02.6439106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:02.6439614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:02.9815160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:02.9824675Z 2022-08-17T13:26:02.9824813Z Running tests... 2022-08-17T13:26:02.9825787Z ---------------------------------------------------------------------- 2022-08-17T13:26:03.2456323Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11782 2022-08-17T13:26:03.2483365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11783 2022-08-17T13:26:03.2510509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11784 2022-08-17T13:26:04.2373005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:04.2373662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:04.2376299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:04.2376886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:04.2541706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:04.2542345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:04.2545967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:04.2546647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:04.2979479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:04.2980524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:04.2983061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:04.2983763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:04.6518585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:04.6619737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:04.6620370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:04.6621080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:04.6621959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:04.6622879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:04.6726704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:04.7632837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:04.7633460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:04.7634950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:26:04.7839964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:26:04.7840448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:26:04.7841163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:04.7841795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:04.7844022Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7844672Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7850290Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7851111Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7858244Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7864705Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7869140Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7869928Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7873962Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7878571Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7881424Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7882278Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7886307Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7888997Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7891369Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7891844Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7895460Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7898303Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7901258Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7901807Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7905238Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7907993Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7910654Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7911514Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7915013Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7917774Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7920483Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7921360Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7924557Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7928833Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7931719Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7932192Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7936153Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7938919Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7941082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:26:04.7942061Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7942370Z 2022-08-17T13:26:04.7946442Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7949149Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7952257Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7952751Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7956934Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7959935Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7963269Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7963581Z 2022-08-17T13:26:04.7967731Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7969301Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:26:04.7972170Z STAGE:2022-08-17 13:26:04 11784:11784 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:04.7972654Z STAGE:2022-08-17 13:26:04 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:26:05.2563161Z ok (2.274s) 2022-08-17T13:26:05.2563377Z 2022-08-17T13:26:05.2563789Z ---------------------------------------------------------------------- 2022-08-17T13:26:05.2564060Z Ran 1 test in 2.274s 2022-08-17T13:26:05.2564189Z 2022-08-17T13:26:05.2564254Z OK 2022-08-17T13:26:05.2564351Z 2022-08-17T13:26:05.2564449Z Generating XML reports... 2022-08-17T13:26:05.2602363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132602.xml 2022-08-17T13:26:06.5366345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:06.5366742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:06.5369422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:06.5369793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:06.8765658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:06.8775563Z 2022-08-17T13:26:06.8775651Z Running tests... 2022-08-17T13:26:06.8776098Z ---------------------------------------------------------------------- 2022-08-17T13:26:07.1439338Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11843 2022-08-17T13:26:07.1467891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11844 2022-08-17T13:26:07.1497200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11845 2022-08-17T13:26:08.1174714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:08.1175432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:08.1177051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:08.1177701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:08.1215188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:08.1215850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:08.1219354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:08.1220048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:08.1448529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:08.1449209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:08.1452766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:08.1453390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:08.5006456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:08.5107416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:08.5107992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:08.5108820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:08.5109968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:08.5110788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:08.5213908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:08.6122275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:08.6122934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:08.9547348Z skip: CUDA is not available. (2.077s) 2022-08-17T13:26:08.9547602Z 2022-08-17T13:26:08.9547941Z ---------------------------------------------------------------------- 2022-08-17T13:26:08.9548273Z Ran 1 test in 2.077s 2022-08-17T13:26:08.9548407Z 2022-08-17T13:26:08.9548485Z OK (skipped=1) 2022-08-17T13:26:08.9548599Z 2022-08-17T13:26:08.9548676Z Generating XML reports... 2022-08-17T13:26:08.9584343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132606.xml 2022-08-17T13:26:10.2370474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:10.2370897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:10.2373356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:10.2373742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:10.5723330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:10.5733613Z 2022-08-17T13:26:10.5734068Z Running tests... 2022-08-17T13:26:10.5734491Z ---------------------------------------------------------------------- 2022-08-17T13:26:10.8228997Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.249s) 2022-08-17T13:26:10.8229559Z 2022-08-17T13:26:10.8229775Z ---------------------------------------------------------------------- 2022-08-17T13:26:10.8230021Z Ran 1 test in 0.249s 2022-08-17T13:26:10.8230142Z 2022-08-17T13:26:10.8230216Z OK (skipped=1) 2022-08-17T13:26:10.8230329Z 2022-08-17T13:26:10.8230418Z Generating XML reports... 2022-08-17T13:26:10.8252906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132610.xml 2022-08-17T13:26:12.0406938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:12.0407352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:12.0410118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:12.0410523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:12.3829059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:12.3839117Z 2022-08-17T13:26:12.3839359Z Running tests... 2022-08-17T13:26:12.3839739Z ---------------------------------------------------------------------- 2022-08-17T13:26:12.3857486Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:26:12.3858058Z 2022-08-17T13:26:12.3858429Z ---------------------------------------------------------------------- 2022-08-17T13:26:12.3858939Z Ran 1 test in 0.002s 2022-08-17T13:26:12.3859058Z 2022-08-17T13:26:12.3859136Z OK (skipped=1) 2022-08-17T13:26:12.3859253Z 2022-08-17T13:26:12.3859343Z Generating XML reports... 2022-08-17T13:26:12.3883648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132612.xml 2022-08-17T13:26:13.5304583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:13.5304985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:13.5307069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:13.5307433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:13.8719579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:13.8729079Z 2022-08-17T13:26:13.8729671Z Running tests... 2022-08-17T13:26:13.8730363Z ---------------------------------------------------------------------- 2022-08-17T13:26:13.8747284Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:26:13.8748117Z 2022-08-17T13:26:13.8748352Z ---------------------------------------------------------------------- 2022-08-17T13:26:13.8748618Z Ran 1 test in 0.002s 2022-08-17T13:26:13.8748750Z 2022-08-17T13:26:13.8748826Z OK (skipped=1) 2022-08-17T13:26:13.8748938Z 2022-08-17T13:26:13.8749013Z Generating XML reports... 2022-08-17T13:26:13.8773778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132613.xml 2022-08-17T13:26:15.0162944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:15.0163980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:15.0185926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:15.0186604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:15.3606782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:15.3617583Z 2022-08-17T13:26:15.3617938Z Running tests... 2022-08-17T13:26:15.3618620Z ---------------------------------------------------------------------- 2022-08-17T13:26:15.6316412Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11926 2022-08-17T13:26:15.6346146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11927 2022-08-17T13:26:15.6376478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11928 2022-08-17T13:26:16.6163934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:16.6164419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:16.6167058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:16.6167717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:16.6398848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:16.6399513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:16.6402843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:16.6403521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:16.6535198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:16.6536154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:16.6538927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:16.6539583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:17.0225703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:17.0226325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:17.0226808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:17.0227778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:17.0228732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:17.0229589Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:17.0331059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:17.1237157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:17.1237897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:17.4425245Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T13:26:17.4425567Z 2022-08-17T13:26:17.4426080Z ---------------------------------------------------------------------- 2022-08-17T13:26:17.4426418Z Ran 1 test in 2.081s 2022-08-17T13:26:17.4426538Z 2022-08-17T13:26:17.4426616Z OK (skipped=1) 2022-08-17T13:26:17.4426732Z 2022-08-17T13:26:17.4426823Z Generating XML reports... 2022-08-17T13:26:17.4464747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132615.xml 2022-08-17T13:26:18.7343858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:18.7344267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:18.7346883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:18.7347377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:19.0745696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:19.0755543Z 2022-08-17T13:26:19.0755681Z Running tests... 2022-08-17T13:26:19.0756307Z ---------------------------------------------------------------------- 2022-08-17T13:26:19.3416870Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11979 2022-08-17T13:26:19.3446172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11980 2022-08-17T13:26:19.3474101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11981 2022-08-17T13:26:20.3247998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:20.3248503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:20.3252058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:20.3252709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:20.3265419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:20.3266068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:20.3270780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:20.3271408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:20.3695162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:20.3695915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:20.3698844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:20.3699412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:20.7379669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:20.7380133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:20.7380766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:20.7381758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:20.7382790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:20.7383591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:20.7485721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:20.8392363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:20.8392970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:21.1523762Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:26:21.1524067Z 2022-08-17T13:26:21.1524535Z ---------------------------------------------------------------------- 2022-08-17T13:26:21.1524912Z Ran 1 test in 2.077s 2022-08-17T13:26:21.1525097Z 2022-08-17T13:26:21.1525215Z OK (skipped=1) 2022-08-17T13:26:21.1525404Z 2022-08-17T13:26:21.1525536Z Generating XML reports... 2022-08-17T13:26:21.1561455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132619.xml 2022-08-17T13:26:22.4619097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:22.4619497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:22.4622012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:22.4622397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:22.8079133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:22.8089343Z 2022-08-17T13:26:22.8089492Z Running tests... 2022-08-17T13:26:22.8089942Z ---------------------------------------------------------------------- 2022-08-17T13:26:23.0764549Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12032 2022-08-17T13:26:23.0793898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12033 2022-08-17T13:26:23.0821975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12034 2022-08-17T13:26:24.0569491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:24.0570118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:24.0571703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:24.0572642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:24.0806112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:24.0806799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:24.0810834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:24.0811545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:24.0930913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:24.0931301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:24.0934972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:24.0935355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:24.4530761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:24.4584150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:24.4584892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:24.4585539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:24.4586083Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:24.4631133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:24.4692040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:24.4692822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:24.5643456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:24.8871688Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T13:26:24.8872088Z 2022-08-17T13:26:24.8872454Z ---------------------------------------------------------------------- 2022-08-17T13:26:24.8872723Z Ran 1 test in 2.078s 2022-08-17T13:26:24.8872846Z 2022-08-17T13:26:24.8872926Z OK (skipped=1) 2022-08-17T13:26:24.8873042Z 2022-08-17T13:26:24.8873119Z Generating XML reports... 2022-08-17T13:26:24.8908335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132622.xml 2022-08-17T13:26:26.1985778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:26.1986440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:26.1989336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:26.1989930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:26.5458849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:26.5468955Z 2022-08-17T13:26:26.5469534Z Running tests... 2022-08-17T13:26:26.5469937Z ---------------------------------------------------------------------- 2022-08-17T13:26:26.8171304Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12085 2022-08-17T13:26:26.8200057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12086 2022-08-17T13:26:26.8229415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12087 2022-08-17T13:26:27.8092507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:27.8093155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:27.8095500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:27.8095888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:27.8208855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:27.8209431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:27.8213453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:27.8214088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:27.8269530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:27.8270193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:27.8273280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:27.8274215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:28.1884245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:28.1884735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:28.1885235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:28.1886446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:28.1887320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:28.1888181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:28.1990229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:28.2896781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:28.2897192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:28.6278614Z skip: Need at least 2 CUDA devices (2.081s) 2022-08-17T13:26:28.6278900Z 2022-08-17T13:26:28.6279255Z ---------------------------------------------------------------------- 2022-08-17T13:26:28.6279519Z Ran 1 test in 2.081s 2022-08-17T13:26:28.6279640Z 2022-08-17T13:26:28.6279716Z OK (skipped=1) 2022-08-17T13:26:28.6279887Z 2022-08-17T13:26:28.6279978Z Generating XML reports... 2022-08-17T13:26:28.6315162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132626.xml 2022-08-17T13:26:29.9231239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:29.9231618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:29.9234412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:29.9234936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:30.2677875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:30.2688061Z 2022-08-17T13:26:30.2688199Z Running tests... 2022-08-17T13:26:30.2688816Z ---------------------------------------------------------------------- 2022-08-17T13:26:30.2720580Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:26:30.2721384Z 2022-08-17T13:26:30.2721989Z ---------------------------------------------------------------------- 2022-08-17T13:26:30.2722488Z Ran 1 test in 0.003s 2022-08-17T13:26:30.2722655Z 2022-08-17T13:26:30.2722734Z OK (skipped=1) 2022-08-17T13:26:30.2722834Z 2022-08-17T13:26:30.2722936Z Generating XML reports... 2022-08-17T13:26:30.2747050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132630.xml 2022-08-17T13:26:31.4368272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:31.4368667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:31.4371276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:31.4371712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:31.7898529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:31.7909855Z 2022-08-17T13:26:31.7910374Z Running tests... 2022-08-17T13:26:31.7910798Z ---------------------------------------------------------------------- 2022-08-17T13:26:32.0512742Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12148 2022-08-17T13:26:32.0540045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12149 2022-08-17T13:26:32.0567764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12150 2022-08-17T13:26:33.0341251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:33.0341864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:33.0344296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:33.0344987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:33.0352177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:33.0352843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:33.0356752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:33.0357323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:33.0459785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:33.0460215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:33.0464154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:33.0464656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:33.4223677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:33.4325383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:33.4325818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:33.4326741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:33.4327328Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:33.4327937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:33.4333603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:33.4335003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:33.4336821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:33.7614681Z skip: Need at least 2 CUDA devices (1.970s) 2022-08-17T13:26:33.7614976Z 2022-08-17T13:26:33.7615501Z ---------------------------------------------------------------------- 2022-08-17T13:26:33.7615975Z Ran 1 test in 1.970s 2022-08-17T13:26:33.7616100Z 2022-08-17T13:26:33.7616175Z OK (skipped=1) 2022-08-17T13:26:33.7616276Z 2022-08-17T13:26:33.7616364Z Generating XML reports... 2022-08-17T13:26:33.7651188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132631.xml 2022-08-17T13:26:35.0758023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:35.0758502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:35.0760826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:35.0761206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:35.4213662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:35.4223747Z 2022-08-17T13:26:35.4223887Z Running tests... 2022-08-17T13:26:35.4224266Z ---------------------------------------------------------------------- 2022-08-17T13:26:35.6923043Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12201 2022-08-17T13:26:35.6950598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12202 2022-08-17T13:26:35.6979193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12203 2022-08-17T13:26:36.6585009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:36.6592136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:36.6593204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:36.6593873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:36.6617798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:36.6618415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:36.6623022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:36.6623735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:36.7072415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:36.7073009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:36.7076156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:36.7076585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:37.0698986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:37.0799801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:37.0800563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:37.0801250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:37.0801998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:37.0802549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:37.0907955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:37.1815248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:37.1815852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:37.5030239Z skip: Need at least 3 CUDA devices (2.080s) 2022-08-17T13:26:37.5030520Z 2022-08-17T13:26:37.5030859Z ---------------------------------------------------------------------- 2022-08-17T13:26:37.5031140Z Ran 1 test in 2.081s 2022-08-17T13:26:37.5031319Z 2022-08-17T13:26:37.5031396Z OK (skipped=1) 2022-08-17T13:26:37.5031497Z 2022-08-17T13:26:37.5031602Z Generating XML reports... 2022-08-17T13:26:37.5067282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132635.xml 2022-08-17T13:26:38.7866653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:38.7867300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:38.7869357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:38.7869761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:39.1293633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:39.1302703Z 2022-08-17T13:26:39.1303143Z Running tests... 2022-08-17T13:26:39.1303569Z ---------------------------------------------------------------------- 2022-08-17T13:26:39.1345324Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s) 2022-08-17T13:26:39.1345826Z 2022-08-17T13:26:39.1346122Z ---------------------------------------------------------------------- 2022-08-17T13:26:39.1346462Z Ran 1 test in 0.004s 2022-08-17T13:26:39.1346594Z 2022-08-17T13:26:39.1346679Z OK (skipped=1) 2022-08-17T13:26:39.1346795Z 2022-08-17T13:26:39.1346885Z Generating XML reports... 2022-08-17T13:26:39.1371585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132639.xml 2022-08-17T13:26:40.2838005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:40.2838400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:40.2840910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:40.2841314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:40.6208464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:40.6218351Z 2022-08-17T13:26:40.6218447Z Running tests... 2022-08-17T13:26:40.6219192Z ---------------------------------------------------------------------- 2022-08-17T13:26:40.6255590Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:26:40.6255878Z 2022-08-17T13:26:40.6256095Z ---------------------------------------------------------------------- 2022-08-17T13:26:40.6256357Z Ran 1 test in 0.004s 2022-08-17T13:26:40.6256463Z 2022-08-17T13:26:40.6256542Z OK (skipped=1) 2022-08-17T13:26:40.6256654Z 2022-08-17T13:26:40.6256744Z Generating XML reports... 2022-08-17T13:26:40.6280871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132640.xml 2022-08-17T13:26:41.7834800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:41.7835184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:41.7838057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:41.7838469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:42.1284100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:42.1294084Z 2022-08-17T13:26:42.1294193Z Running tests... 2022-08-17T13:26:42.1294790Z ---------------------------------------------------------------------- 2022-08-17T13:26:42.3954521Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12274 2022-08-17T13:26:42.3984156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12275 2022-08-17T13:26:42.4013745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12276 2022-08-17T13:26:43.3881156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:43.3881858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:43.3883988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:43.3884396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:43.4020016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:43.4020700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:43.4024907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:43.4025597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:43.4228705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:43.4229734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:43.4233270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:43.4233931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:43.7974650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:43.8076371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:43.8077026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:43.8078031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:43.8078660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:43.8079460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:43.8085284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:43.8085823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:43.8086680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:43.8154979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwp6j66c 2022-08-17T13:26:43.8155734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffk2l8gz 2022-08-17T13:26:43.8156651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcob5c42 2022-08-17T13:26:43.8157234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwp6j66c/_remote_module_non_scriptable.py 2022-08-17T13:26:43.8157806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffk2l8gz/_remote_module_non_scriptable.py 2022-08-17T13:26:43.8158235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcob5c42/_remote_module_non_scriptable.py 2022-08-17T13:26:43.8270697Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8272043Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8274029Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8275491Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:26:43.8276355Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:26:43.8277722Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:26:43.8278500Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:26:43.8279639Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:26:43.8280463Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:26:43.8282269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:26:43.8282697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:26:43.8283085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:26:43.8285322Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8286796Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8287996Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8292156Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8294239Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8296209Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8298045Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8300010Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8301725Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8303231Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8304413Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8305799Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8307373Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8308957Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:43.8310744Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:26:44.1060902Z ok (1.976s) 2022-08-17T13:26:44.1061155Z 2022-08-17T13:26:44.1061653Z ---------------------------------------------------------------------- 2022-08-17T13:26:44.1062167Z Ran 1 test in 1.977s 2022-08-17T13:26:44.1062378Z 2022-08-17T13:26:44.1062488Z OK 2022-08-17T13:26:44.1062661Z 2022-08-17T13:26:44.1062893Z Generating XML reports... 2022-08-17T13:26:44.1097851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132642.xml 2022-08-17T13:26:45.3781873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:45.3782274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:45.3784283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:45.3784662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:45.7173414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:45.7183976Z 2022-08-17T13:26:45.7184244Z Running tests... 2022-08-17T13:26:45.7184619Z ---------------------------------------------------------------------- 2022-08-17T13:26:45.7229430Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s) 2022-08-17T13:26:45.7229939Z 2022-08-17T13:26:45.7230219Z ---------------------------------------------------------------------- 2022-08-17T13:26:45.7230466Z Ran 1 test in 0.004s 2022-08-17T13:26:45.7230588Z 2022-08-17T13:26:45.7230666Z OK (skipped=1) 2022-08-17T13:26:45.7230782Z 2022-08-17T13:26:45.7230874Z Generating XML reports... 2022-08-17T13:26:45.7255396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132645.xml 2022-08-17T13:26:46.8706657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:46.8707084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:46.8709476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:46.8709891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:47.2110692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:47.2120079Z 2022-08-17T13:26:47.2120452Z Running tests... 2022-08-17T13:26:47.2120865Z ---------------------------------------------------------------------- 2022-08-17T13:26:47.2150095Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:26:47.2150555Z 2022-08-17T13:26:47.2150977Z ---------------------------------------------------------------------- 2022-08-17T13:26:47.2151357Z Ran 1 test in 0.003s 2022-08-17T13:26:47.2151479Z 2022-08-17T13:26:47.2151555Z OK (skipped=1) 2022-08-17T13:26:47.2151669Z 2022-08-17T13:26:47.2151760Z Generating XML reports... 2022-08-17T13:26:47.2176307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132647.xml 2022-08-17T13:26:48.3684230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:48.3684867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:48.3687710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:48.3688109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:48.7086374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:48.7096213Z 2022-08-17T13:26:48.7096324Z Running tests... 2022-08-17T13:26:48.7097146Z ---------------------------------------------------------------------- 2022-08-17T13:26:48.9749323Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12347 2022-08-17T13:26:48.9777303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12348 2022-08-17T13:26:48.9805572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12349 2022-08-17T13:26:49.9716684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:49.9717087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:49.9720416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:49.9721103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:50.0054365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:50.0054772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:50.0056737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:50.0057312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:50.0183139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:50.0183688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:50.0186933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:50.0187306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:50.4185371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:50.4285412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:50.4286692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:50.4288025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:50.4289050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:50.4290114Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:50.4294569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:50.4295177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:50.4295550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:50.7854105Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T13:26:50.7854440Z 2022-08-17T13:26:50.7854901Z ---------------------------------------------------------------------- 2022-08-17T13:26:50.7855162Z Ran 1 test in 2.076s 2022-08-17T13:26:50.7855283Z 2022-08-17T13:26:50.7855362Z OK (skipped=1) 2022-08-17T13:26:50.7855476Z 2022-08-17T13:26:50.7855552Z Generating XML reports... 2022-08-17T13:26:50.7890612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132648.xml 2022-08-17T13:26:52.0708989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:52.0709698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:52.0712265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:52.0712847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:52.4112328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:52.4122520Z 2022-08-17T13:26:52.4122992Z Running tests... 2022-08-17T13:26:52.4123403Z ---------------------------------------------------------------------- 2022-08-17T13:26:52.6714146Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12400 2022-08-17T13:26:52.6742186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12401 2022-08-17T13:26:52.6770392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12402 2022-08-17T13:26:53.6588402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:53.6588796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:53.6592296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:53.6592995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:53.6627317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:53.6627943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:53.6631663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:53.6632335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:53.6924430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:53.6925095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:53.6928204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:53.6929166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:54.0587673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:54.0688242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:54.0688908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:54.0689675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:54.0690235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:54.0690777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:54.0797705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:54.1703159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:54.1703766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:54.4821909Z skip: Need at least 3 CUDA devices (2.070s) 2022-08-17T13:26:54.4822210Z 2022-08-17T13:26:54.4823044Z ---------------------------------------------------------------------- 2022-08-17T13:26:54.4858243Z Ran 1 test in 2.070s 2022-08-17T13:26:54.4858465Z 2022-08-17T13:26:54.4858600Z OK (skipped=1) 2022-08-17T13:26:54.4858767Z 2022-08-17T13:26:54.4858921Z Generating XML reports... 2022-08-17T13:26:54.4859857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132652.xml 2022-08-17T13:26:55.7871760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:55.7872164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:55.7874180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:55.7874562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:56.1301905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:56.1312247Z 2022-08-17T13:26:56.1312349Z Running tests... 2022-08-17T13:26:56.1313075Z ---------------------------------------------------------------------- 2022-08-17T13:26:56.3954714Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12453 2022-08-17T13:26:56.3983469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12454 2022-08-17T13:26:56.4012351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12455 2022-08-17T13:26:57.3876464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:57.3877160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:57.3878628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:57.3879256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:57.4036816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:57.4037461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:57.4040241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:57.4040918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:57.4514400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:57.4515304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:57.4518226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:57.4518667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:57.8152818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:26:57.8153942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:57.8154390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:26:57.8154777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:26:57.8155307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:57.8155853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:26:57.8258136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:26:57.9167752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:26:57.9168360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:26:58.2060301Z skip: Need at least 3 CUDA devices (2.074s) 2022-08-17T13:26:58.2060649Z 2022-08-17T13:26:58.2061143Z ---------------------------------------------------------------------- 2022-08-17T13:26:58.2061604Z Ran 1 test in 2.075s 2022-08-17T13:26:58.2061819Z 2022-08-17T13:26:58.2061959Z OK (skipped=1) 2022-08-17T13:26:58.2062163Z 2022-08-17T13:26:58.2062318Z Generating XML reports... 2022-08-17T13:26:58.2096502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132656.xml 2022-08-17T13:26:59.4868826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:26:59.4869240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:26:59.4871764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:26:59.4872143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:26:59.8260780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:26:59.8270136Z 2022-08-17T13:26:59.8270240Z Running tests... 2022-08-17T13:26:59.8270699Z ---------------------------------------------------------------------- 2022-08-17T13:27:00.0876312Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12506 2022-08-17T13:27:00.0903355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12507 2022-08-17T13:27:00.0931191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12508 2022-08-17T13:27:01.0666631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:01.0667054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:01.0669951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:01.0670632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:01.0747436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:01.0747821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:01.0751201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:01.0751736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:01.0956520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:01.0957021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:01.0960408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:01.0960910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:01.4638767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:01.4739249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:01.4740074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:01.4740784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:01.4741521Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:01.4742078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:01.4747998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:01.4750564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:01.4751205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:01.7982214Z skip: Need at least 3 CUDA devices (1.971s) 2022-08-17T13:27:01.7982509Z 2022-08-17T13:27:01.7983141Z ---------------------------------------------------------------------- 2022-08-17T13:27:01.7983415Z Ran 1 test in 1.971s 2022-08-17T13:27:01.7983538Z 2022-08-17T13:27:01.7983600Z OK (skipped=1) 2022-08-17T13:27:01.7983715Z 2022-08-17T13:27:01.7983805Z Generating XML reports... 2022-08-17T13:27:01.8018897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132659.xml 2022-08-17T13:27:03.1329505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:03.1330123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:03.1332453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:03.1333055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:03.4784326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:03.4794280Z 2022-08-17T13:27:03.4794778Z Running tests... 2022-08-17T13:27:03.4795206Z ---------------------------------------------------------------------- 2022-08-17T13:27:03.7461530Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12559 2022-08-17T13:27:03.7489196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12560 2022-08-17T13:27:03.7520190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12561 2022-08-17T13:27:04.7230065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:04.7230660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:04.7231176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:04.7232009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:04.7232946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:04.7233568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:04.7234052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:04.7234404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:04.7309802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:04.7310162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:04.7314143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:04.7314503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:05.0910709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:05.1012149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:05.1013457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:05.1014168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:05.1015385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:05.1016258Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:05.1020402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:05.1021165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:05.1022143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:05.4565334Z skip: Need at least 3 CUDA devices (1.977s) 2022-08-17T13:27:05.4565619Z 2022-08-17T13:27:05.4566293Z ---------------------------------------------------------------------- 2022-08-17T13:27:05.4566695Z Ran 1 test in 1.977s 2022-08-17T13:27:05.4566880Z 2022-08-17T13:27:05.4567011Z OK (skipped=1) 2022-08-17T13:27:05.4567192Z 2022-08-17T13:27:05.4567339Z Generating XML reports... 2022-08-17T13:27:05.4602906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132703.xml 2022-08-17T13:27:06.7386136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:06.7386525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:06.7389300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:06.7389712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:07.0851103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:07.0860445Z 2022-08-17T13:27:07.0860552Z Running tests... 2022-08-17T13:27:07.0861037Z ---------------------------------------------------------------------- 2022-08-17T13:27:07.3480212Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12612 2022-08-17T13:27:07.3509077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12613 2022-08-17T13:27:07.3537081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12614 2022-08-17T13:27:08.4159722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:08.4160695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:08.4161506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:08.4162127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:08.4178555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:08.4179167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:08.4182915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:08.4183820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:08.4188674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:08.4189772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:08.4193849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:08.4194523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:08.7878004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:08.7979043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:08.7979791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:08.7980874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:08.7981809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:08.7982378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:08.7987246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:08.7989780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:08.7990169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:09.1589569Z skip: Need at least 3 CUDA devices (2.073s) 2022-08-17T13:27:09.1589870Z 2022-08-17T13:27:09.1590196Z ---------------------------------------------------------------------- 2022-08-17T13:27:09.1590519Z Ran 1 test in 2.073s 2022-08-17T13:27:09.1590627Z 2022-08-17T13:27:09.1590707Z OK (skipped=1) 2022-08-17T13:27:09.1590820Z 2022-08-17T13:27:09.1590909Z Generating XML reports... 2022-08-17T13:27:09.1626275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132707.xml 2022-08-17T13:27:10.4687044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:10.4687426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:10.4689932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:10.4690395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:10.8120743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:10.8130493Z 2022-08-17T13:27:10.8130749Z Running tests... 2022-08-17T13:27:10.8131397Z ---------------------------------------------------------------------- 2022-08-17T13:27:11.0800441Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12665 2022-08-17T13:27:11.0828520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12666 2022-08-17T13:27:11.0857508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12667 2022-08-17T13:27:12.0357752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:12.0358395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:12.0360795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:12.0361423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:12.0841156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:12.0841806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:12.0844520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:12.0845114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:12.0858875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:12.0859291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:12.0863349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:12.0863849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:12.4498360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:12.4588703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:12.4589253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:12.4590006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:12.4590649Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:12.4598847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:12.4696201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:12.4697001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:12.5612448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:12.8905823Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T13:27:12.8906104Z 2022-08-17T13:27:12.8906415Z ---------------------------------------------------------------------- 2022-08-17T13:27:12.8906701Z Ran 1 test in 2.077s 2022-08-17T13:27:12.8906824Z 2022-08-17T13:27:12.8906903Z OK (skipped=1) 2022-08-17T13:27:12.8907016Z 2022-08-17T13:27:12.8907108Z Generating XML reports... 2022-08-17T13:27:12.8945859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132710.xml 2022-08-17T13:27:14.1895243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:14.1895674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:14.1897916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:14.1898296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:14.5354531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:14.5364166Z 2022-08-17T13:27:14.5364275Z Running tests... 2022-08-17T13:27:14.5364709Z ---------------------------------------------------------------------- 2022-08-17T13:27:14.8055668Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12718 2022-08-17T13:27:14.8084559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12719 2022-08-17T13:27:14.8113712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12720 2022-08-17T13:27:15.7726675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:15.7727073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:15.7730078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:15.7730597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:15.8138353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:15.8139026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:15.8139812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:15.8140144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:15.8143251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:15.8143867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:15.8144688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:15.8145292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:16.1915325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:16.1926928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:16.1927352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:16.1928134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:16.1928761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:16.2016674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:16.2035198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:16.2035789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:16.3029770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:16.6161181Z skip: Need at least 2 CUDA devices (2.079s) 2022-08-17T13:27:16.6161579Z 2022-08-17T13:27:16.6162278Z ---------------------------------------------------------------------- 2022-08-17T13:27:16.6162681Z Ran 1 test in 2.080s 2022-08-17T13:27:16.6162807Z 2022-08-17T13:27:16.6162898Z OK (skipped=1) 2022-08-17T13:27:16.6163012Z 2022-08-17T13:27:16.6163101Z Generating XML reports... 2022-08-17T13:27:16.6198099Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132714.xml 2022-08-17T13:27:17.9015361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:17.9015750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:17.9018449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:17.9018834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:18.2448646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:18.2458485Z 2022-08-17T13:27:18.2458601Z Running tests... 2022-08-17T13:27:18.2459103Z ---------------------------------------------------------------------- 2022-08-17T13:27:18.5090891Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12771 2022-08-17T13:27:18.5118285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12772 2022-08-17T13:27:18.5146983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12773 2022-08-17T13:27:19.4838601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:19.4839288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:19.4843117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:19.4843858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:19.5250389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:19.5251063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:19.5253111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:19.5253774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:19.5345313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:19.5349588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:19.5350427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:19.5351093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:19.9028993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:19.9033294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:19.9034046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:19.9034743Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:19.9035281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:19.9130501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:19.9141511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:19.9141962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:20.0142810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:20.3195607Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:27:20.3195914Z 2022-08-17T13:27:20.3196461Z ---------------------------------------------------------------------- 2022-08-17T13:27:20.3196887Z Ran 1 test in 2.074s 2022-08-17T13:27:20.3197008Z 2022-08-17T13:27:20.3197084Z OK (skipped=1) 2022-08-17T13:27:20.3197197Z 2022-08-17T13:27:20.3197272Z Generating XML reports... 2022-08-17T13:27:20.3232061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132718.xml 2022-08-17T13:27:21.6139209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:21.6139610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:21.6142274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:21.6142737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:21.9519287Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:21.9528670Z 2022-08-17T13:27:21.9528811Z Running tests... 2022-08-17T13:27:21.9529435Z ---------------------------------------------------------------------- 2022-08-17T13:27:22.2143428Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12824 2022-08-17T13:27:22.2170437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12825 2022-08-17T13:27:22.2198405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12826 2022-08-17T13:27:23.2157676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:23.2158230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:23.2158811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:23.2159250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:23.2160050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:23.2160425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:23.2161884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:23.2162556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:23.2401095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:23.2401561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:23.2404041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:23.2404581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:23.6059926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:23.6161176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:23.6161722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:23.6162393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:23.6162950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:23.6163501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:23.6169499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:23.6170043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:23.6172392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:23.9248563Z skip: Need at least 2 CUDA devices (1.972s) 2022-08-17T13:27:23.9248791Z 2022-08-17T13:27:23.9249278Z ---------------------------------------------------------------------- 2022-08-17T13:27:23.9249719Z Ran 1 test in 1.972s 2022-08-17T13:27:23.9249842Z 2022-08-17T13:27:23.9249921Z OK (skipped=1) 2022-08-17T13:27:23.9250037Z 2022-08-17T13:27:23.9250151Z Generating XML reports... 2022-08-17T13:27:23.9284945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132721.xml 2022-08-17T13:27:25.2195177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:25.2195585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:25.2197438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:25.2197804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:25.5620530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:25.5630712Z 2022-08-17T13:27:25.5630823Z Running tests... 2022-08-17T13:27:25.5631259Z ---------------------------------------------------------------------- 2022-08-17T13:27:25.8255961Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12877 2022-08-17T13:27:25.8285659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12878 2022-08-17T13:27:25.8312962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12879 2022-08-17T13:27:26.8135254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:26.8135909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:26.8138648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:26.8139217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:26.8151801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:26.8152408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:26.8156129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:26.8156738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:26.8230395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:26.8231049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:26.8234460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:26.8234869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:27.1807507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:27.1908336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:27.1908872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:27.1909552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:27.1910118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:27.1910655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:27.1917230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:27.1917844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:27.1918920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:27.5362037Z skip: Need at least 2 CUDA devices (1.973s) 2022-08-17T13:27:27.5362330Z 2022-08-17T13:27:27.5362784Z ---------------------------------------------------------------------- 2022-08-17T13:27:27.5363194Z Ran 1 test in 1.973s 2022-08-17T13:27:27.5363397Z 2022-08-17T13:27:27.5363519Z OK (skipped=1) 2022-08-17T13:27:27.5363695Z 2022-08-17T13:27:27.5363832Z Generating XML reports... 2022-08-17T13:27:27.5398747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132725.xml 2022-08-17T13:27:28.8090616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:28.8091325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:28.8094124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:28.8094999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:29.1531953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:29.1542676Z 2022-08-17T13:27:29.1543033Z Running tests... 2022-08-17T13:27:29.1543686Z ---------------------------------------------------------------------- 2022-08-17T13:27:29.4195740Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12930 2022-08-17T13:27:29.4223834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12931 2022-08-17T13:27:29.4251439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12932 2022-08-17T13:27:30.4437658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:30.4438340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:30.4440788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:30.4441266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:30.4577541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:30.4577926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:30.4582142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:30.4582938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:30.4763452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:30.4764110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:30.4767427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:30.4768095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:30.8448181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:30.8549466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:30.8550143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:30.8551150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:30.8551926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:30.8552739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:30.8558782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:30.8559726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:30.8560191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:31.2304983Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:27:31.2305289Z 2022-08-17T13:27:31.2305813Z ---------------------------------------------------------------------- 2022-08-17T13:27:31.2306288Z Ran 1 test in 2.076s 2022-08-17T13:27:31.2306541Z 2022-08-17T13:27:31.2306656Z OK (skipped=1) 2022-08-17T13:27:31.2306858Z 2022-08-17T13:27:31.2307011Z Generating XML reports... 2022-08-17T13:27:31.2341159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132729.xml 2022-08-17T13:27:32.5358082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:32.5358499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:32.5360786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:32.5361164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:32.8795154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:32.8805230Z 2022-08-17T13:27:32.8805383Z Running tests... 2022-08-17T13:27:32.8805819Z ---------------------------------------------------------------------- 2022-08-17T13:27:33.1458864Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12983 2022-08-17T13:27:33.1487253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12984 2022-08-17T13:27:33.1514961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12985 2022-08-17T13:27:34.1591364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:34.1592016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:34.1594695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:34.1595343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:34.1659274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:34.1659878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:34.1664254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:34.1664711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:34.1752665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:34.1753338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:34.1756479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:34.1757005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:34.5331624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:34.5432235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:34.5432717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:34.5433351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:34.5433896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:34.5434450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:34.5440629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:34.5441226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:34.5442042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:34.8563811Z skip: Need at least 2 CUDA devices (1.976s) 2022-08-17T13:27:34.8564142Z 2022-08-17T13:27:34.8564932Z ---------------------------------------------------------------------- 2022-08-17T13:27:34.8565406Z Ran 1 test in 1.976s 2022-08-17T13:27:34.8565606Z 2022-08-17T13:27:34.8565739Z OK (skipped=1) 2022-08-17T13:27:34.8566108Z 2022-08-17T13:27:34.8566252Z Generating XML reports... 2022-08-17T13:27:34.8599469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132732.xml 2022-08-17T13:27:36.1103486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:36.1103900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:36.1106146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:36.1106563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:36.4483168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:36.4493234Z 2022-08-17T13:27:36.4493395Z Running tests... 2022-08-17T13:27:36.4493867Z ---------------------------------------------------------------------- 2022-08-17T13:27:36.7091176Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13036 2022-08-17T13:27:36.7118588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13037 2022-08-17T13:27:36.7146339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13038 2022-08-17T13:27:37.7341935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:37.7342702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:37.7345359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:37.7345985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:37.7704785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:37.7705452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:37.7708702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:37.7709349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:37.7947268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:37.7948104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:37.7950198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:37.7950761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:38.1601302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:38.1702196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:38.1702935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:38.1703965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:38.1704582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:38.1705245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:38.1711587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:38.1712342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:38.1713401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:38.5199935Z skip: Need at least 2 CUDA devices (2.070s) 2022-08-17T13:27:38.5200149Z 2022-08-17T13:27:38.5200543Z ---------------------------------------------------------------------- 2022-08-17T13:27:38.5200803Z Ran 1 test in 2.071s 2022-08-17T13:27:38.5200934Z 2022-08-17T13:27:38.5201016Z OK (skipped=1) 2022-08-17T13:27:38.5201199Z 2022-08-17T13:27:38.5201292Z Generating XML reports... 2022-08-17T13:27:38.5236296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132736.xml 2022-08-17T13:27:39.7823663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:39.7824081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:39.7826556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:39.7827048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:40.1304169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:40.1314388Z 2022-08-17T13:27:40.1314513Z Running tests... 2022-08-17T13:27:40.1315116Z ---------------------------------------------------------------------- 2022-08-17T13:27:40.4033354Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13089 2022-08-17T13:27:40.4062401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13090 2022-08-17T13:27:40.4090001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13091 2022-08-17T13:27:41.4112347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:41.4113053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:41.4113831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:41.4114432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:41.4115158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:41.4115759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:41.4116862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:41.4117431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:41.4154899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:41.4155534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:41.4159055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:41.4159754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:41.7895896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:41.7997899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:41.7998645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:41.7999077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:41.7999777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:41.8000319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:41.8006620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:41.8007277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:41.8007845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:42.1135272Z skip: Need at least 2 CUDA devices (1.982s) 2022-08-17T13:27:42.1135678Z 2022-08-17T13:27:42.1136172Z ---------------------------------------------------------------------- 2022-08-17T13:27:42.1136434Z Ran 1 test in 1.982s 2022-08-17T13:27:42.1136557Z 2022-08-17T13:27:42.1136632Z OK (skipped=1) 2022-08-17T13:27:42.1136746Z 2022-08-17T13:27:42.1136823Z Generating XML reports... 2022-08-17T13:27:42.1174182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132740.xml 2022-08-17T13:27:43.3835771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:43.3838487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:43.3839060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:43.3839442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:43.7244550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:43.7254562Z 2022-08-17T13:27:43.7254758Z Running tests... 2022-08-17T13:27:43.7255169Z ---------------------------------------------------------------------- 2022-08-17T13:27:43.9887804Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13142 2022-08-17T13:27:43.9917084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13143 2022-08-17T13:27:43.9945973Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13144 2022-08-17T13:27:44.9885741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:44.9886331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:44.9888943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:44.9889653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:44.9944150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:44.9944635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:44.9948362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:44.9948826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:45.0015834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:45.0016476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:45.0019614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:45.0020302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:45.3561923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:45.3621308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:45.3621711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:45.3622365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:45.3622996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:45.3663605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:45.3729413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:45.3730093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:45.4676308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:45.7992727Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:27:45.7993198Z 2022-08-17T13:27:45.7993831Z ---------------------------------------------------------------------- 2022-08-17T13:27:45.8001818Z Ran 1 test in 2.074s 2022-08-17T13:27:45.8001959Z 2022-08-17T13:27:45.8002044Z OK (skipped=1) 2022-08-17T13:27:45.8002159Z 2022-08-17T13:27:45.8002237Z Generating XML reports... 2022-08-17T13:27:45.8029512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132743.xml 2022-08-17T13:27:47.1042383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:47.1043010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:47.1044687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:47.1045342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:47.4442080Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:47.4452578Z 2022-08-17T13:27:47.4452928Z Running tests... 2022-08-17T13:27:47.4453566Z ---------------------------------------------------------------------- 2022-08-17T13:27:47.7067839Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13195 2022-08-17T13:27:47.7095455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13196 2022-08-17T13:27:47.7123207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13197 2022-08-17T13:27:48.6950827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:48.6951286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:48.6951830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:48.6952247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:48.6954061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:48.6954432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:48.6957598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:48.6958264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:48.7373257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:48.7373960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:48.7375632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:48.7376326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:49.1083218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:49.1083910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:49.1084309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:49.1084959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:49.1085545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:49.1086257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:49.1189518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:49.2096421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:49.2097109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:49.5173881Z skip: Need at least 2 CUDA devices (2.072s) 2022-08-17T13:27:49.5174206Z 2022-08-17T13:27:49.5174691Z ---------------------------------------------------------------------- 2022-08-17T13:27:49.5175092Z Ran 1 test in 2.072s 2022-08-17T13:27:49.5175278Z 2022-08-17T13:27:49.5175400Z OK (skipped=1) 2022-08-17T13:27:49.5175608Z 2022-08-17T13:27:49.5175727Z Generating XML reports... 2022-08-17T13:27:49.5211207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132747.xml 2022-08-17T13:27:50.7970837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:50.7971324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:50.7973500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:50.7973878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:51.1368790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:51.1378814Z 2022-08-17T13:27:51.1378983Z Running tests... 2022-08-17T13:27:51.1379341Z ---------------------------------------------------------------------- 2022-08-17T13:27:51.4043922Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13248 2022-08-17T13:27:51.4071783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13249 2022-08-17T13:27:51.4101002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13250 2022-08-17T13:27:52.3773731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:52.3774137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:52.3777248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:52.3777960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:52.3778460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:52.3778827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:52.3782410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:52.3783190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:52.4247635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:52.4248009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:52.4251705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:52.4252229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:52.7844421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:52.7844902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:52.7845304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:52.7846705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:52.7847472Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:52.7848020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:52.7852318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:52.7853235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:52.7854159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:53.1147706Z skip: Need at least 2 CUDA devices (1.977s) 2022-08-17T13:27:53.1148069Z 2022-08-17T13:27:53.1148433Z ---------------------------------------------------------------------- 2022-08-17T13:27:53.1148710Z Ran 1 test in 1.977s 2022-08-17T13:27:53.1148831Z 2022-08-17T13:27:53.1148908Z OK (skipped=1) 2022-08-17T13:27:53.1149025Z 2022-08-17T13:27:53.1149115Z Generating XML reports... 2022-08-17T13:27:53.1183279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132751.xml 2022-08-17T13:27:54.4022871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:54.4023273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:54.4025523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:54.4025902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:54.7392942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:54.7402419Z 2022-08-17T13:27:54.7402569Z Running tests... 2022-08-17T13:27:54.7403135Z ---------------------------------------------------------------------- 2022-08-17T13:27:54.7422329Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:27:54.7422872Z 2022-08-17T13:27:54.7423224Z ---------------------------------------------------------------------- 2022-08-17T13:27:54.7423483Z Ran 1 test in 0.002s 2022-08-17T13:27:54.7423606Z 2022-08-17T13:27:54.7423683Z OK (skipped=1) 2022-08-17T13:27:54.7423796Z 2022-08-17T13:27:54.7423885Z Generating XML reports... 2022-08-17T13:27:54.7446793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132754.xml 2022-08-17T13:27:55.8792962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:55.8793358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:55.8796097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:55.8796470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:56.2179700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:56.2190008Z 2022-08-17T13:27:56.2190323Z Running tests... 2022-08-17T13:27:56.2191009Z ---------------------------------------------------------------------- 2022-08-17T13:27:56.4836633Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13311 2022-08-17T13:27:56.4864701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13312 2022-08-17T13:27:56.4893306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13313 2022-08-17T13:27:57.4658062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:57.4658663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:57.4660959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:57.4661552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:57.4737954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:57.4738605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:57.4741853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:57.4742610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:57.4948202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:57.4948867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:57.4950841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:57.4951532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:57.8690482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:27:57.8791547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:27:57.8792343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:27:57.8793106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:57.8793892Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:57.8794436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:27:57.8799797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:27:57.8801039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:27:57.8802137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:27:58.1940102Z skip: Need at least 2 CUDA devices (1.975s) 2022-08-17T13:27:58.1940400Z 2022-08-17T13:27:58.1940927Z ---------------------------------------------------------------------- 2022-08-17T13:27:58.1941285Z Ran 1 test in 1.975s 2022-08-17T13:27:58.1941487Z 2022-08-17T13:27:58.1941600Z OK (skipped=1) 2022-08-17T13:27:58.1941721Z 2022-08-17T13:27:58.1943781Z Generating XML reports... 2022-08-17T13:27:58.1978005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132756.xml 2022-08-17T13:27:59.4848301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:27:59.4848703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:27:59.4850712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:27:59.4851090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:27:59.8274944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:27:59.8284634Z 2022-08-17T13:27:59.8284740Z Running tests... 2022-08-17T13:27:59.8285270Z ---------------------------------------------------------------------- 2022-08-17T13:28:00.0950588Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13364 2022-08-17T13:28:00.0979601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13365 2022-08-17T13:28:00.1007445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13366 2022-08-17T13:28:01.0847038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:01.0847428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:01.0850878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:01.0851359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:01.0959249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:01.0959718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:01.0963309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:01.0963695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:01.1153088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:01.1153614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:01.1157574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:01.1158122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:01.4859593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:01.4961135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:01.4961715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:01.4962405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:01.4963009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:01.4963903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:01.4969753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:01.4970408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:01.4971494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:01.8053374Z skip: Need at least 2 CUDA devices (1.976s) 2022-08-17T13:28:01.8053609Z 2022-08-17T13:28:01.8054134Z ---------------------------------------------------------------------- 2022-08-17T13:28:01.8054547Z Ran 1 test in 1.977s 2022-08-17T13:28:01.8054679Z 2022-08-17T13:28:01.8054963Z OK (skipped=1) 2022-08-17T13:28:01.8055078Z 2022-08-17T13:28:01.8055170Z Generating XML reports... 2022-08-17T13:28:01.8089276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132759.xml 2022-08-17T13:28:03.0816190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:03.0816589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:03.0818514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:03.0819087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:03.4201617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:03.4211435Z 2022-08-17T13:28:03.4211582Z Running tests... 2022-08-17T13:28:03.4212234Z ---------------------------------------------------------------------- 2022-08-17T13:28:03.6909851Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13417 2022-08-17T13:28:03.6951501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13418 2022-08-17T13:28:03.6997638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13419 2022-08-17T13:28:04.6702187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:04.6702919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:04.6704565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:04.6705161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:04.6761341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:04.6761954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:04.6766416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:04.6767025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:04.6981988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:04.6982740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:04.6986722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:04.6987607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:05.0555260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:05.0655616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:05.0656223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:05.0656854Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:05.0657414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:05.0657966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:05.0762497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:05.0829899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp7sm6hr 2022-08-17T13:28:05.0832132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp7sm6hr/_remote_module_non_scriptable.py 2022-08-17T13:28:05.1668151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:05.1668670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:05.1740806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36muundz 2022-08-17T13:28:05.1743198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36muundz/_remote_module_non_scriptable.py 2022-08-17T13:28:05.1743632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl61gv3yc 2022-08-17T13:28:05.1745092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl61gv3yc/_remote_module_non_scriptable.py 2022-08-17T13:28:05.1877001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:28:05.1877434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:28:05.1877821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:28:05.7052657Z ok (2.284s) 2022-08-17T13:28:05.7052907Z 2022-08-17T13:28:05.7053417Z ---------------------------------------------------------------------- 2022-08-17T13:28:05.7053686Z Ran 1 test in 2.284s 2022-08-17T13:28:05.7053810Z 2022-08-17T13:28:05.7053875Z OK 2022-08-17T13:28:05.7053973Z 2022-08-17T13:28:05.7054056Z Generating XML reports... 2022-08-17T13:28:05.7090135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132803.xml 2022-08-17T13:28:07.0034712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:07.0035127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:07.0037987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:07.0038413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:07.3410721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:07.3420287Z 2022-08-17T13:28:07.3420736Z Running tests... 2022-08-17T13:28:07.3421384Z ---------------------------------------------------------------------- 2022-08-17T13:28:07.6049565Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13479 2022-08-17T13:28:07.6079070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13480 2022-08-17T13:28:07.6106823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13481 2022-08-17T13:28:08.6017223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:08.6017913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:08.6021307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:08.6021940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:08.6223791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:08.6224429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:08.6226965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:08.6227616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:08.6334474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:08.6334874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:08.6338753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:08.6339129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:08.9972812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:09.0073568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:09.0074145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:09.0074827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:09.0075736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:09.0076560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:09.0181165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:09.1087078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:09.1087720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:09.4154287Z skip: CUDA is not available. (2.073s) 2022-08-17T13:28:09.4154538Z 2022-08-17T13:28:09.4154861Z ---------------------------------------------------------------------- 2022-08-17T13:28:09.4155209Z Ran 1 test in 2.073s 2022-08-17T13:28:09.4155334Z 2022-08-17T13:28:09.4155410Z OK (skipped=1) 2022-08-17T13:28:09.4155529Z 2022-08-17T13:28:09.4155623Z Generating XML reports... 2022-08-17T13:28:09.4191366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132807.xml 2022-08-17T13:28:10.6918934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:10.6921574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:10.6922058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:10.6922427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:11.0361773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:11.0372800Z 2022-08-17T13:28:11.0373213Z Running tests... 2022-08-17T13:28:11.0373861Z ---------------------------------------------------------------------- 2022-08-17T13:28:11.0395791Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:28:11.0396151Z 2022-08-17T13:28:11.0396514Z ---------------------------------------------------------------------- 2022-08-17T13:28:11.0397000Z Ran 1 test in 0.002s 2022-08-17T13:28:11.0397209Z 2022-08-17T13:28:11.0397325Z OK (skipped=1) 2022-08-17T13:28:11.0397525Z 2022-08-17T13:28:11.0397679Z Generating XML reports... 2022-08-17T13:28:11.0422732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132811.xml 2022-08-17T13:28:12.2160169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:12.2160550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:12.2163035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:12.2163598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:12.5598059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:12.5608011Z 2022-08-17T13:28:12.5608136Z Running tests... 2022-08-17T13:28:12.5609159Z ---------------------------------------------------------------------- 2022-08-17T13:28:12.5630028Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:28:12.5630467Z 2022-08-17T13:28:12.5630764Z ---------------------------------------------------------------------- 2022-08-17T13:28:12.5631008Z Ran 1 test in 0.002s 2022-08-17T13:28:12.5631128Z 2022-08-17T13:28:12.5631203Z OK (skipped=1) 2022-08-17T13:28:12.5631317Z 2022-08-17T13:28:12.5631407Z Generating XML reports... 2022-08-17T13:28:12.5655672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132812.xml 2022-08-17T13:28:13.7227068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:13.7227759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:13.7230717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:13.7231302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:14.0660299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:14.0670843Z 2022-08-17T13:28:14.0671160Z Running tests... 2022-08-17T13:28:14.0671807Z ---------------------------------------------------------------------- 2022-08-17T13:28:14.0689916Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:28:14.0690253Z 2022-08-17T13:28:14.0690591Z ---------------------------------------------------------------------- 2022-08-17T13:28:14.0691042Z Ran 1 test in 0.002s 2022-08-17T13:28:14.0691271Z 2022-08-17T13:28:14.0691388Z OK (skipped=1) 2022-08-17T13:28:14.0691505Z 2022-08-17T13:28:14.0691589Z Generating XML reports... 2022-08-17T13:28:14.0715825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132814.xml 2022-08-17T13:28:15.2293453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:15.2294092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:15.2294922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:15.2295583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:15.5698427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:15.5709335Z 2022-08-17T13:28:15.5709617Z Running tests... 2022-08-17T13:28:15.5710243Z ---------------------------------------------------------------------- 2022-08-17T13:28:15.5729003Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:28:15.5729310Z 2022-08-17T13:28:15.5729602Z ---------------------------------------------------------------------- 2022-08-17T13:28:15.5729890Z Ran 1 test in 0.002s 2022-08-17T13:28:15.5730082Z 2022-08-17T13:28:15.5730225Z OK (skipped=1) 2022-08-17T13:28:15.5730437Z 2022-08-17T13:28:15.5730600Z Generating XML reports... 2022-08-17T13:28:15.5756329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132815.xml 2022-08-17T13:28:16.7412121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:16.7412518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:16.7415230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:16.7415619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:17.0849840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:17.0859774Z 2022-08-17T13:28:17.0860170Z Running tests... 2022-08-17T13:28:17.0860601Z ---------------------------------------------------------------------- 2022-08-17T13:28:17.0886527Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:28:17.0887013Z 2022-08-17T13:28:17.0887404Z ---------------------------------------------------------------------- 2022-08-17T13:28:17.0887673Z Ran 1 test in 0.003s 2022-08-17T13:28:17.0887795Z 2022-08-17T13:28:17.0887872Z OK (skipped=1) 2022-08-17T13:28:17.0887988Z 2022-08-17T13:28:17.0888079Z Generating XML reports... 2022-08-17T13:28:17.0912212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132817.xml 2022-08-17T13:28:18.2503354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:18.2503760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:18.2506252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:18.2506797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:18.5949301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:18.5959020Z 2022-08-17T13:28:18.5959165Z Running tests... 2022-08-17T13:28:18.5959778Z ---------------------------------------------------------------------- 2022-08-17T13:28:18.8599440Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13582 2022-08-17T13:28:18.8628251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13583 2022-08-17T13:28:18.8656007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13584 2022-08-17T13:28:19.8908545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:19.8909241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:19.8910068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:19.8910692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:19.9103689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:19.9104338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:19.9107148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:19.9107807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:19.9556300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:19.9556948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:19.9559651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:19.9560244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:20.3185376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:20.3215989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:20.3216427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:20.3217359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:20.3217912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:20.3286853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:20.3323282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:20.3324407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:20.4300355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:20.7708829Z skip: Need at least 2 CUDA devices (2.175s) 2022-08-17T13:28:20.7709125Z 2022-08-17T13:28:20.7709656Z ---------------------------------------------------------------------- 2022-08-17T13:28:20.7709934Z Ran 1 test in 2.175s 2022-08-17T13:28:20.7710054Z 2022-08-17T13:28:20.7710131Z OK (skipped=1) 2022-08-17T13:28:20.7710245Z 2022-08-17T13:28:20.7710334Z Generating XML reports... 2022-08-17T13:28:20.7745724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132818.xml 2022-08-17T13:28:22.0780707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:22.0781106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:22.0783091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:22.0783628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:22.4185032Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:22.4195174Z 2022-08-17T13:28:22.4195442Z Running tests... 2022-08-17T13:28:22.4195920Z ---------------------------------------------------------------------- 2022-08-17T13:28:22.6799780Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13635 2022-08-17T13:28:22.6827296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13636 2022-08-17T13:28:22.6854896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13637 2022-08-17T13:28:23.6914599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:23.6915608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:23.6917371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:23.6918021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:23.7028164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:23.7028795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:23.7032431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:23.7033092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:23.7114701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:23.7115362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:23.7119089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:23.7119768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:24.0741137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:24.0841519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:24.0842166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:24.0843203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:24.0843832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:24.0844639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:24.0849781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:24.0850849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:24.0852022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:24.3904652Z skip: Need at least 2 CUDA devices (1.971s) 2022-08-17T13:28:24.3905019Z 2022-08-17T13:28:24.3905376Z ---------------------------------------------------------------------- 2022-08-17T13:28:24.3905650Z Ran 1 test in 1.971s 2022-08-17T13:28:24.3905773Z 2022-08-17T13:28:24.3905848Z OK (skipped=1) 2022-08-17T13:28:24.3905948Z 2022-08-17T13:28:24.3906038Z Generating XML reports... 2022-08-17T13:28:24.3940441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132822.xml 2022-08-17T13:28:25.6499762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:25.6500167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:25.6501985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:25.6502518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:25.9958265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:25.9968147Z 2022-08-17T13:28:25.9968409Z Running tests... 2022-08-17T13:28:25.9969039Z ---------------------------------------------------------------------- 2022-08-17T13:28:25.9986191Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:28:25.9986659Z 2022-08-17T13:28:25.9987157Z ---------------------------------------------------------------------- 2022-08-17T13:28:25.9987420Z Ran 1 test in 0.002s 2022-08-17T13:28:25.9987541Z 2022-08-17T13:28:25.9987623Z OK (skipped=1) 2022-08-17T13:28:25.9987738Z 2022-08-17T13:28:25.9987813Z Generating XML reports... 2022-08-17T13:28:26.0012878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132825.xml 2022-08-17T13:28:27.1516353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:27.1516760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:27.1519116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:27.1519492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:27.5001995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:27.5012830Z 2022-08-17T13:28:27.5013233Z Running tests... 2022-08-17T13:28:27.5013677Z ---------------------------------------------------------------------- 2022-08-17T13:28:27.5033551Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:28:27.5033924Z 2022-08-17T13:28:27.5034165Z ---------------------------------------------------------------------- 2022-08-17T13:28:27.5034430Z Ran 1 test in 0.002s 2022-08-17T13:28:27.5034552Z 2022-08-17T13:28:27.5034653Z OK (skipped=1) 2022-08-17T13:28:27.5034825Z 2022-08-17T13:28:27.5034901Z Generating XML reports... 2022-08-17T13:28:27.5059429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132827.xml 2022-08-17T13:28:28.6622003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:28.6622419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:28.6624786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:28.6625173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:29.0025393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:29.0034559Z 2022-08-17T13:28:29.0034921Z Running tests... 2022-08-17T13:28:29.0035351Z ---------------------------------------------------------------------- 2022-08-17T13:28:29.2653576Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13708 2022-08-17T13:28:29.2682250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13709 2022-08-17T13:28:29.2710955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13710 2022-08-17T13:28:30.2581646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:30.2582348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:30.2585000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:30.2585407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:30.3013829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:30.3014511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:30.3015306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:30.3015859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:30.3017168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:30.3018103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:30.3018856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:30.3019503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:30.6792928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:30.6893785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:30.6894305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:30.6894953Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:30.6895506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:30.6896085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:30.7002822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:30.7003462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:30.7907610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:31.0758271Z skip: Need at least 2 CUDA devices (2.072s) 2022-08-17T13:28:31.0758588Z 2022-08-17T13:28:31.0759133Z ---------------------------------------------------------------------- 2022-08-17T13:28:31.0759481Z Ran 1 test in 2.072s 2022-08-17T13:28:31.0759589Z 2022-08-17T13:28:31.0759671Z OK (skipped=1) 2022-08-17T13:28:31.0759788Z 2022-08-17T13:28:31.0759877Z Generating XML reports... 2022-08-17T13:28:31.0795465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132828.xml 2022-08-17T13:28:32.3765228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:32.3765688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:32.3768635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:32.3769097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:32.7205054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:32.7215105Z 2022-08-17T13:28:32.7215269Z Running tests... 2022-08-17T13:28:32.7215918Z ---------------------------------------------------------------------- 2022-08-17T13:28:32.9822204Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13761 2022-08-17T13:28:32.9849327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13762 2022-08-17T13:28:32.9876932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13763 2022-08-17T13:28:33.9783030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:33.9783709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:33.9786739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:33.9787345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:33.9798986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:33.9799612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:33.9803635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:33.9804310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:33.9850168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:33.9850788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:33.9854291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:33.9854942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:34.3500647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:34.3564547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:34.3565035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:34.3565686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:34.3566641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:34.3601989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:34.3672764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:34.3673484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:34.4614636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:34.7925028Z skip: Need at least 2 CUDA devices (2.071s) 2022-08-17T13:28:34.7925332Z 2022-08-17T13:28:34.7925803Z ---------------------------------------------------------------------- 2022-08-17T13:28:34.7926264Z Ran 1 test in 2.071s 2022-08-17T13:28:34.7926386Z 2022-08-17T13:28:34.7926462Z OK (skipped=1) 2022-08-17T13:28:34.7926576Z 2022-08-17T13:28:34.7926652Z Generating XML reports... 2022-08-17T13:28:34.7961949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132832.xml 2022-08-17T13:28:36.0880089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:36.0880483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:36.0883088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:36.0883464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:36.4290889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:36.4301230Z 2022-08-17T13:28:36.4301643Z Running tests... 2022-08-17T13:28:36.4302099Z ---------------------------------------------------------------------- 2022-08-17T13:28:36.4327918Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:28:36.4328263Z 2022-08-17T13:28:36.4328478Z ---------------------------------------------------------------------- 2022-08-17T13:28:36.4328724Z Ran 1 test in 0.003s 2022-08-17T13:28:36.4328843Z 2022-08-17T13:28:36.4328920Z OK (skipped=1) 2022-08-17T13:28:36.4329033Z 2022-08-17T13:28:36.4329121Z Generating XML reports... 2022-08-17T13:28:36.4355441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132836.xml 2022-08-17T13:28:37.5939487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:37.5940376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:37.5946778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:37.5947189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:37.9437066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:37.9446934Z 2022-08-17T13:28:37.9447065Z Running tests... 2022-08-17T13:28:37.9448160Z ---------------------------------------------------------------------- 2022-08-17T13:28:38.2141972Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13824 2022-08-17T13:28:38.2171709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13825 2022-08-17T13:28:38.2199617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13826 2022-08-17T13:28:39.2064160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:39.2065483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:39.2066828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:39.2067435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:39.2212418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:39.2212905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:39.2214896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:39.2215282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:39.2380466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:39.2381130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:39.2382994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:39.2383634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:39.6098635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:39.6099106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:39.6099574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:39.6100305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:39.6100868Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:39.6101490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:39.6206963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:39.6207576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:39.7112006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:40.0251606Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T13:28:40.0251810Z 2022-08-17T13:28:40.0252182Z ---------------------------------------------------------------------- 2022-08-17T13:28:40.0252431Z Ran 1 test in 2.080s 2022-08-17T13:28:40.0252550Z 2022-08-17T13:28:40.0252627Z OK (skipped=1) 2022-08-17T13:28:40.0252990Z 2022-08-17T13:28:40.0253083Z Generating XML reports... 2022-08-17T13:28:40.0287043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132837.xml 2022-08-17T13:28:41.3371112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:41.3371519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:41.3373142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:41.3373593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:41.6836573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:41.6846761Z 2022-08-17T13:28:41.6846907Z Running tests... 2022-08-17T13:28:41.6847336Z ---------------------------------------------------------------------- 2022-08-17T13:28:41.9533257Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13877 2022-08-17T13:28:41.9561937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13878 2022-08-17T13:28:41.9591577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13879 2022-08-17T13:28:42.9522195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:42.9522866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:42.9525568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:42.9526161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:42.9587799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:42.9588198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:42.9591825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:42.9592210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:42.9694326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:42.9694931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:42.9699029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:42.9699673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:43.3442391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:43.3443110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:43.3443772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:43.3444212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:43.3444733Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:43.3445287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:43.3450341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:43.3451172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:43.3452010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:43.6638368Z skip: Need at least 2 CUDA devices (1.979s) 2022-08-17T13:28:43.6638654Z 2022-08-17T13:28:43.6639197Z ---------------------------------------------------------------------- 2022-08-17T13:28:43.6639498Z Ran 1 test in 1.979s 2022-08-17T13:28:43.6639619Z 2022-08-17T13:28:43.6639696Z OK (skipped=1) 2022-08-17T13:28:43.6639810Z 2022-08-17T13:28:43.6639920Z Generating XML reports... 2022-08-17T13:28:43.6674585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132841.xml 2022-08-17T13:28:44.9199206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:44.9199592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:44.9202657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:44.9203037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:45.2626563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:45.2637468Z 2022-08-17T13:28:45.2637910Z Running tests... 2022-08-17T13:28:45.2638313Z ---------------------------------------------------------------------- 2022-08-17T13:28:45.5296569Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13930 2022-08-17T13:28:45.5325275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13931 2022-08-17T13:28:45.5353244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13932 2022-08-17T13:28:46.5116362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:46.5117049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:46.5120884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:46.5121543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:46.5133736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:46.5134307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:46.5138205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:46.5138861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:46.5141213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:46.5141818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:46.5146960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:46.5147627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:46.8930712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:46.8931372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:46.8931977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:46.8932909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:46.8933484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:46.8934358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:46.8939854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:46.8940650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:46.9943815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:47.3402231Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:28:47.3402485Z 2022-08-17T13:28:47.3402788Z ---------------------------------------------------------------------- 2022-08-17T13:28:47.3403050Z Ran 1 test in 2.076s 2022-08-17T13:28:47.3403170Z 2022-08-17T13:28:47.3403245Z OK (skipped=1) 2022-08-17T13:28:47.3403360Z 2022-08-17T13:28:47.3403447Z Generating XML reports... 2022-08-17T13:28:47.3442134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132845.xml 2022-08-17T13:28:48.6212440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:48.6213126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:48.6228081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:48.6228708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:48.9665809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:48.9676114Z 2022-08-17T13:28:48.9676401Z Running tests... 2022-08-17T13:28:48.9677101Z ---------------------------------------------------------------------- 2022-08-17T13:28:49.2352706Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13983 2022-08-17T13:28:49.2381895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13984 2022-08-17T13:28:49.2411671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13985 2022-08-17T13:28:50.2446185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:50.2446913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:50.2450223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:50.2450933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:50.2775516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:50.2776198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:50.2779067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:50.2779770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:50.3043583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:50.3044146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:50.3047328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:50.3047872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:50.6658195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:50.6759536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:50.6760688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:50.6761453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:50.6762502Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:50.6763054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:50.6767923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:50.6768504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:50.6771101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:51.0460911Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T13:28:51.0461167Z 2022-08-17T13:28:51.0461684Z ---------------------------------------------------------------------- 2022-08-17T13:28:51.0462112Z Ran 1 test in 2.078s 2022-08-17T13:28:51.0462232Z 2022-08-17T13:28:51.0462296Z OK (skipped=1) 2022-08-17T13:28:51.0462409Z 2022-08-17T13:28:51.0464254Z Generating XML reports... 2022-08-17T13:28:51.0498727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132848.xml 2022-08-17T13:28:52.3594114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:52.3594760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:52.3597467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:52.3597852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:52.7031212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:52.7041238Z 2022-08-17T13:28:52.7041392Z Running tests... 2022-08-17T13:28:52.7041797Z ---------------------------------------------------------------------- 2022-08-17T13:28:52.9687946Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14036 2022-08-17T13:28:52.9716862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14037 2022-08-17T13:28:52.9745197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14038 2022-08-17T13:28:54.0243151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:54.0243685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:54.0244404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:54.0244907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:54.0245697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:54.0246528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:54.0247400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:54.0247868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:54.0355770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:54.0356131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:54.0359601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:54.0359990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:54.4124533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:54.4125146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:54.4125746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:54.4126594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:54.4127149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:54.4127699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:54.4230734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:54.5137246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:54.5137870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:54.7794133Z skip: Need at least 2 CUDA devices (2.075s) 2022-08-17T13:28:54.7794487Z 2022-08-17T13:28:54.7794962Z ---------------------------------------------------------------------- 2022-08-17T13:28:54.7795227Z Ran 1 test in 2.075s 2022-08-17T13:28:54.7795348Z 2022-08-17T13:28:54.7795426Z OK (skipped=1) 2022-08-17T13:28:54.7795541Z 2022-08-17T13:28:54.7795631Z Generating XML reports... 2022-08-17T13:28:54.7829821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132852.xml 2022-08-17T13:28:56.0389373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:56.0389789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:56.0392149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:56.0392663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:56.3831098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:56.3840924Z 2022-08-17T13:28:56.3841180Z Running tests... 2022-08-17T13:28:56.3841789Z ---------------------------------------------------------------------- 2022-08-17T13:28:56.6483088Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14089 2022-08-17T13:28:56.6512409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14090 2022-08-17T13:28:56.6540345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14091 2022-08-17T13:28:57.6326901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:57.6327469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:57.6328808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:57.6329333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:57.6408405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:57.6408909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:57.6412475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:57.6412977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:57.6496954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:57.6497331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:57.6501684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:57.6502308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:58.0066840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:28:58.0109569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:28:58.0110157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:28:58.0110813Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:58.0111367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:58.0169127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:28:58.0218201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:28:58.0218848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:28:58.1181502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:28:58.3587423Z skip: Need at least 2 CUDA devices (1.974s) 2022-08-17T13:28:58.3587714Z 2022-08-17T13:28:58.3588314Z ---------------------------------------------------------------------- 2022-08-17T13:28:58.3588598Z Ran 1 test in 1.974s 2022-08-17T13:28:58.3588719Z 2022-08-17T13:28:58.3588798Z OK (skipped=1) 2022-08-17T13:28:58.3588915Z 2022-08-17T13:28:58.3588993Z Generating XML reports... 2022-08-17T13:28:58.3623405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132856.xml 2022-08-17T13:28:59.6364565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:28:59.6364965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:28:59.6367386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:28:59.6367956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:28:59.9760216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:28:59.9769827Z 2022-08-17T13:28:59.9769962Z Running tests... 2022-08-17T13:28:59.9770571Z ---------------------------------------------------------------------- 2022-08-17T13:29:00.2377042Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s) 2022-08-17T13:29:00.2377674Z 2022-08-17T13:29:00.2377879Z ---------------------------------------------------------------------- 2022-08-17T13:29:00.2378155Z Ran 1 test in 0.260s 2022-08-17T13:29:00.2378278Z 2022-08-17T13:29:00.2378354Z OK (skipped=1) 2022-08-17T13:29:00.2378469Z 2022-08-17T13:29:00.2378557Z Generating XML reports... 2022-08-17T13:29:00.2404241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132859.xml 2022-08-17T13:29:01.5148201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:01.5148601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:01.5151090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:01.5151472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:01.8611324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:01.8621542Z 2022-08-17T13:29:01.8621877Z Running tests... 2022-08-17T13:29:01.8652029Z ---------------------------------------------------------------------- 2022-08-17T13:29:01.8653071Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:29:01.8653427Z 2022-08-17T13:29:01.8653640Z ---------------------------------------------------------------------- 2022-08-17T13:29:01.8653894Z Ran 1 test in 0.003s 2022-08-17T13:29:01.8654002Z 2022-08-17T13:29:01.8654079Z OK (skipped=1) 2022-08-17T13:29:01.8654193Z 2022-08-17T13:29:01.8654283Z Generating XML reports... 2022-08-17T13:29:01.8678187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132901.xml 2022-08-17T13:29:03.0249360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:03.0249791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:03.0252543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:03.0253196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:03.3677352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:03.3687586Z 2022-08-17T13:29:03.3687925Z Running tests... 2022-08-17T13:29:03.3688559Z ---------------------------------------------------------------------- 2022-08-17T13:29:03.6368728Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14162 2022-08-17T13:29:03.6396156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14163 2022-08-17T13:29:03.6423867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14164 2022-08-17T13:29:04.6217054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:04.6217440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:04.6220430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:04.6220813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:04.6436052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:04.6436406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:04.6440400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:04.6441085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:04.6511719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:04.6512396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:04.6516239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:04.6516863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:05.0269279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:05.0370708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:05.0371249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:05.0372124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:05.0373254Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:05.0373827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:05.0379545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:05.0380767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:05.0381599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:05.0592026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:05.0592445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:05.0592822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:05.0593466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:05.0594030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:05.0594743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:05.4472425Z ok (2.078s) 2022-08-17T13:29:05.4472663Z 2022-08-17T13:29:05.4473196Z ---------------------------------------------------------------------- 2022-08-17T13:29:05.4473599Z Ran 1 test in 2.078s 2022-08-17T13:29:05.4473727Z 2022-08-17T13:29:05.4473793Z OK 2022-08-17T13:29:05.4473891Z 2022-08-17T13:29:05.4473990Z Generating XML reports... 2022-08-17T13:29:05.4512691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132903.xml 2022-08-17T13:29:06.7664176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:06.7664578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:06.7666746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:06.7667111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:07.1087989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:07.1097980Z 2022-08-17T13:29:07.1098170Z Running tests... 2022-08-17T13:29:07.1098546Z ---------------------------------------------------------------------- 2022-08-17T13:29:07.3746798Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14224 2022-08-17T13:29:07.3777004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14225 2022-08-17T13:29:07.3805216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14226 2022-08-17T13:29:08.3823372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:08.3824116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:08.3826971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:08.3827560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:08.3875863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:08.3876539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:08.3880172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:08.3880839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:08.4479418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:08.4480118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:08.4483449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:08.4484057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:08.8134407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:08.8235509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:08.8236208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:08.8237050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:08.8237617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:08.8238336Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:08.8343355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:08.9249911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:08.9250716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:08.9252406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:08.9455851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:08.9456658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:08.9457399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:08.9457964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:08.9458523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:09.3858053Z ok (2.276s) 2022-08-17T13:29:09.3858259Z 2022-08-17T13:29:09.3858699Z ---------------------------------------------------------------------- 2022-08-17T13:29:09.3859091Z Ran 1 test in 2.276s 2022-08-17T13:29:09.3859214Z 2022-08-17T13:29:09.3859270Z OK 2022-08-17T13:29:09.3859369Z 2022-08-17T13:29:09.3859467Z Generating XML reports... 2022-08-17T13:29:09.3894644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132907.xml 2022-08-17T13:29:10.6652059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:10.6652544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:10.6655355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:10.6655924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:11.0169429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:11.0178906Z 2022-08-17T13:29:11.0179054Z Running tests... 2022-08-17T13:29:11.0179694Z ---------------------------------------------------------------------- 2022-08-17T13:29:11.2876638Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14283 2022-08-17T13:29:11.2905516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14284 2022-08-17T13:29:11.2935509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14285 2022-08-17T13:29:12.2803945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:12.2804612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:12.2806372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:12.2806953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:12.2909058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:12.2909455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:12.2912294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:12.2912674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:12.3013868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:12.3014438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:12.3018414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:12.3018883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:12.6591349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:12.6675839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:12.6676275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:12.6676930Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:12.6677485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:12.6692107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:12.6783791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:12.6784689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:12.7705589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:13.0988291Z skip: Need at least 2 CUDA devices (2.081s) 2022-08-17T13:29:13.0988596Z 2022-08-17T13:29:13.0988996Z ---------------------------------------------------------------------- 2022-08-17T13:29:13.0989294Z Ran 1 test in 2.081s 2022-08-17T13:29:13.0989414Z 2022-08-17T13:29:13.0989489Z OK (skipped=1) 2022-08-17T13:29:13.0989602Z 2022-08-17T13:29:13.0989678Z Generating XML reports... 2022-08-17T13:29:13.1024378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132911.xml 2022-08-17T13:29:14.4145436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:14.4146049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:14.4148279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:14.4148892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:14.7624828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:14.7635397Z 2022-08-17T13:29:14.7635743Z Running tests... 2022-08-17T13:29:14.7636365Z ---------------------------------------------------------------------- 2022-08-17T13:29:14.7657880Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:29:14.7658288Z 2022-08-17T13:29:14.7658509Z ---------------------------------------------------------------------- 2022-08-17T13:29:14.7658782Z Ran 1 test in 0.002s 2022-08-17T13:29:14.7658906Z 2022-08-17T13:29:14.7658968Z OK (skipped=1) 2022-08-17T13:29:14.7659079Z 2022-08-17T13:29:14.7659167Z Generating XML reports... 2022-08-17T13:29:14.7684147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132914.xml 2022-08-17T13:29:15.9344410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:15.9344821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:15.9347336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:15.9347833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:16.2780032Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:16.2789926Z 2022-08-17T13:29:16.2790027Z Running tests... 2022-08-17T13:29:16.2790934Z ---------------------------------------------------------------------- 2022-08-17T13:29:16.5427427Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14346 2022-08-17T13:29:16.5455116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14347 2022-08-17T13:29:16.5482739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14348 2022-08-17T13:29:17.5457695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:17.5458388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:17.5459324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:17.5459933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:17.5461338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:17.5461752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:17.5462231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:17.5462604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:17.5617668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:17.5618067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:17.5621332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:17.5622026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:17.9278048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:17.9380048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:17.9381113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:17.9381812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:17.9382364Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:17.9383374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:17.9387600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:17.9388767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:17.9389715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:18.2530649Z ok (1.974s) 2022-08-17T13:29:18.2530930Z 2022-08-17T13:29:18.2531464Z ---------------------------------------------------------------------- 2022-08-17T13:29:18.2531753Z Ran 1 test in 1.974s 2022-08-17T13:29:18.2531876Z 2022-08-17T13:29:18.2531940Z OK 2022-08-17T13:29:18.2532035Z 2022-08-17T13:29:18.2532119Z Generating XML reports... 2022-08-17T13:29:18.2566142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132916.xml 2022-08-17T13:29:19.5317996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:19.5320356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:19.5321350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:19.5321725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:19.8699768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:19.8709207Z 2022-08-17T13:29:19.8709671Z Running tests... 2022-08-17T13:29:19.8710229Z ---------------------------------------------------------------------- 2022-08-17T13:29:20.1362830Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14399 2022-08-17T13:29:20.1389827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14400 2022-08-17T13:29:20.1421644Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14401 2022-08-17T13:29:21.1360770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:21.1361445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:21.1364822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:21.1365438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:21.1437594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:21.1438010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:21.1441888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:21.1442413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:21.1849797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:21.1850196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:21.1853690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:21.1854085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:21.5506545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:21.5606660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:21.5607282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:21.5608239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:21.5609151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:21.5609711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:21.5715792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:21.5716231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:21.5719913Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.5720540Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6619855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:21.6624388Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6631220Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6632279Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6633030Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6637220Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6637749Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6642707Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6646425Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6648009Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6651832Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6653137Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6656218Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6659781Z STAGE:2022-08-17 13:29:21 14399:14399 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6667352Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:21.6670115Z STAGE:2022-08-17 13:29:21 14400:14400 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:21.6672312Z STAGE:2022-08-17 13:29:21 14401:14401 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:22.1476442Z ok (2.276s) 2022-08-17T13:29:22.1476719Z 2022-08-17T13:29:22.1477229Z ---------------------------------------------------------------------- 2022-08-17T13:29:22.1477699Z Ran 1 test in 2.277s 2022-08-17T13:29:22.1477903Z 2022-08-17T13:29:22.1478017Z OK 2022-08-17T13:29:22.1478197Z 2022-08-17T13:29:22.1478360Z Generating XML reports... 2022-08-17T13:29:22.1512681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132919.xml 2022-08-17T13:29:23.4606480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:23.4606882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:23.4609293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:23.4610018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:23.8049384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:23.8059272Z 2022-08-17T13:29:23.8059414Z Running tests... 2022-08-17T13:29:23.8059936Z ---------------------------------------------------------------------- 2022-08-17T13:29:24.0718916Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14455 2022-08-17T13:29:24.0747440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14456 2022-08-17T13:29:24.0775929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14457 2022-08-17T13:29:25.0759412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:25.0760079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:25.0761278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:25.0761904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:25.0779300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:25.0779917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:25.0783621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:25.0784256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:25.0823561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:25.0824156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:25.0828100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:25.0828748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:25.4441250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:25.4540841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:25.4541467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:25.4542303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:25.4542933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:25.4543486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:25.4648991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:25.5554109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:25.8825965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:25.8826528Z ok (2.076s) 2022-08-17T13:29:25.8826704Z 2022-08-17T13:29:25.8827024Z ---------------------------------------------------------------------- 2022-08-17T13:29:25.8827272Z Ran 1 test in 2.076s 2022-08-17T13:29:25.8827399Z 2022-08-17T13:29:25.8827466Z OK 2022-08-17T13:29:25.8827562Z 2022-08-17T13:29:25.8827658Z Generating XML reports... 2022-08-17T13:29:25.8862040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132923.xml 2022-08-17T13:29:27.1877324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:27.1878018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:27.1879337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:27.1879721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:27.5254668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:27.5263548Z 2022-08-17T13:29:27.5263685Z Running tests... 2022-08-17T13:29:27.5264536Z ---------------------------------------------------------------------- 2022-08-17T13:29:27.5281818Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:29:27.5282279Z 2022-08-17T13:29:27.5282706Z ---------------------------------------------------------------------- 2022-08-17T13:29:27.5282973Z Ran 1 test in 0.002s 2022-08-17T13:29:27.5283094Z 2022-08-17T13:29:27.5283191Z OK (skipped=1) 2022-08-17T13:29:27.5283307Z 2022-08-17T13:29:27.5283397Z Generating XML reports... 2022-08-17T13:29:27.5307585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132927.xml 2022-08-17T13:29:28.6588379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:28.6588800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:28.6590757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:28.6591131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:28.9945031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:28.9954211Z 2022-08-17T13:29:28.9954315Z Running tests... 2022-08-17T13:29:28.9954921Z ---------------------------------------------------------------------- 2022-08-17T13:29:29.2547249Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14518 2022-08-17T13:29:29.2575890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14519 2022-08-17T13:29:29.2604139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14520 2022-08-17T13:29:30.2414299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:30.2414684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:30.2416989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:30.2417493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:30.2491062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:30.2491548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:30.2495516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:30.2496040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:30.2522319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:30.2522786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:30.2527432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:30.2528063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:30.6169433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:30.6270775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:30.6271686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:30.6272634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:30.6273610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:30.6274212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:30.6377761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:30.7283870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:30.7284327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:30.7492154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:30.7594044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:30.7594861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:30.7595838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:30.7596813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:30.7597656Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:30.7598566Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7598872Z 2022-08-17T13:29:30.7599133Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7604870Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7605605Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7606565Z STAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7610818Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7611529Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7614654Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7617970Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7623628Z STAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7627578Z STAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7628368Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7631713Z STAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7635095Z STAGE:2022-08-17 13:29:30 14518:14518 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7641036Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:30.7644336Z STAGE:2022-08-17 13:29:30 14519:14519 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:30.7645117Z STAGE:2022-08-17 13:29:30 14520:14520 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:31.2655013Z ok (2.270s) 2022-08-17T13:29:31.2655235Z 2022-08-17T13:29:31.2655618Z ---------------------------------------------------------------------- 2022-08-17T13:29:31.2655868Z Ran 1 test in 2.270s 2022-08-17T13:29:31.2656013Z 2022-08-17T13:29:31.2656078Z OK 2022-08-17T13:29:31.2656193Z 2022-08-17T13:29:31.2656351Z Generating XML reports... 2022-08-17T13:29:31.2691205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132928.xml 2022-08-17T13:29:32.5519404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:32.5519841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:32.5522538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:32.5523067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:32.8978736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:32.8988868Z 2022-08-17T13:29:32.8988999Z Running tests... 2022-08-17T13:29:32.8989626Z ---------------------------------------------------------------------- 2022-08-17T13:29:33.1696448Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14583 2022-08-17T13:29:33.1725767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14584 2022-08-17T13:29:33.1755977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14585 2022-08-17T13:29:34.1559891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:34.1560291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:34.1562550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:34.1562976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:34.1753896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:34.1754339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:34.1757407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:34.1757768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:34.2195619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:34.2196158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:34.2198986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:34.2199549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:34.5773363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:34.5773881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:34.5774275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:34.5774920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:34.5775479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:34.5776032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:34.5880088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:34.5881093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:34.6086491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:34.6086893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:34.6784375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:34.6787102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:34.6787786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:34.6796228Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:34.6797071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:34.6799262Z STAGE:2022-08-17 13:29:34 14584:14584 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:34.6899973Z STAGE:2022-08-17 13:29:34 14585:14585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:34.6904612Z STAGE:2022-08-17 13:29:34 14585:14585 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:34.6905778Z STAGE:2022-08-17 13:29:34 14584:14584 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:34.6911098Z STAGE:2022-08-17 13:29:34 14585:14585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:34.6920391Z STAGE:2022-08-17 13:29:34 14584:14584 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:29:34.6924018Z STAGE:2022-08-17 13:29:34 14584:14584 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:34.6925238Z STAGE:2022-08-17 13:29:34 14585:14585 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:29:35.1807449Z ok (2.281s) 2022-08-17T13:29:35.1807722Z 2022-08-17T13:29:35.1808187Z ---------------------------------------------------------------------- 2022-08-17T13:29:35.1808577Z Ran 1 test in 2.282s 2022-08-17T13:29:35.1808703Z 2022-08-17T13:29:35.1808767Z OK 2022-08-17T13:29:35.1808883Z 2022-08-17T13:29:35.1809032Z Generating XML reports... 2022-08-17T13:29:35.1845061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132932.xml 2022-08-17T13:29:36.4923553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:36.4924244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:36.4927404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:36.4928044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:36.8402234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:36.8412637Z 2022-08-17T13:29:36.8413089Z Running tests... 2022-08-17T13:29:36.8413563Z ---------------------------------------------------------------------- 2022-08-17T13:29:37.1075469Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14644 2022-08-17T13:29:37.1104790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14645 2022-08-17T13:29:37.1133391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14646 2022-08-17T13:29:38.0826833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:38.0827786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:38.0830408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:38.0830809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:38.0896319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:38.0896732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:38.0899926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:38.0900286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:38.0985857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:38.0986420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:38.0989937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:38.0990395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:38.4559169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:38.4589257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:38.4589679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:38.4590326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:38.4590886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:38.4660160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:38.4698130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:38.4698738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:38.5671629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:38.9184529Z ok (2.077s) 2022-08-17T13:29:38.9184834Z 2022-08-17T13:29:38.9185333Z ---------------------------------------------------------------------- 2022-08-17T13:29:38.9185604Z Ran 1 test in 2.077s 2022-08-17T13:29:38.9185713Z 2022-08-17T13:29:38.9185778Z OK 2022-08-17T13:29:38.9185876Z 2022-08-17T13:29:38.9185974Z Generating XML reports... 2022-08-17T13:29:38.9225224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132936.xml 2022-08-17T13:29:40.2195452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:40.2195919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:40.2198379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:40.2198779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:40.5593178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:40.5603502Z 2022-08-17T13:29:40.5603984Z Running tests... 2022-08-17T13:29:40.5604400Z ---------------------------------------------------------------------- 2022-08-17T13:29:40.8167708Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s) 2022-08-17T13:29:40.8168540Z 2022-08-17T13:29:40.8168773Z ---------------------------------------------------------------------- 2022-08-17T13:29:40.8169030Z Ran 1 test in 0.256s 2022-08-17T13:29:40.8169137Z 2022-08-17T13:29:40.8169219Z OK (skipped=1) 2022-08-17T13:29:40.8169334Z 2022-08-17T13:29:40.8169424Z Generating XML reports... 2022-08-17T13:29:40.8192073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132940.xml 2022-08-17T13:29:42.0724383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:42.0724869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:42.0727512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:42.0727910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:42.4107409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:42.4117196Z 2022-08-17T13:29:42.4117529Z Running tests... 2022-08-17T13:29:42.4118161Z ---------------------------------------------------------------------- 2022-08-17T13:29:42.6735117Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14707 2022-08-17T13:29:42.6762823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14708 2022-08-17T13:29:42.6791697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14709 2022-08-17T13:29:43.6752991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:43.6753651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:43.6756314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:43.6756923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:43.7234879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:43.7235283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:43.7237974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:43.7238456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:43.7260461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:43.7261051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:43.7265121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:43.7265491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:44.1037280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:44.1037744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:44.1038142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:44.1038776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:44.1039331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:44.1039880Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:44.1045389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:44.1046425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:44.1047563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:44.1048159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:44.1253693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:44.1254397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:44.1255014Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:44.1255572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:44.1352891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:44.5842732Z ok (2.172s) 2022-08-17T13:29:44.5843018Z 2022-08-17T13:29:44.5843792Z ---------------------------------------------------------------------- 2022-08-17T13:29:44.5844052Z Ran 1 test in 2.173s 2022-08-17T13:29:44.5844173Z 2022-08-17T13:29:44.5844239Z OK 2022-08-17T13:29:44.5844338Z 2022-08-17T13:29:44.5844434Z Generating XML reports... 2022-08-17T13:29:44.5878942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132942.xml 2022-08-17T13:29:45.8698756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:45.8699149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:45.8701985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:45.8702404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:46.2104097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:46.2114729Z 2022-08-17T13:29:46.2114854Z Running tests... 2022-08-17T13:29:46.2115371Z ---------------------------------------------------------------------- 2022-08-17T13:29:46.4778778Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14766 2022-08-17T13:29:46.4806497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14767 2022-08-17T13:29:46.4835191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14768 2022-08-17T13:29:47.5329768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:47.5330490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:47.5333657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:47.5334084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:47.5341117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:47.5341573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:47.5346909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:47.5347510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:47.5398971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:47.5399584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:47.5403369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:47.5404036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:47.8976537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:47.9062126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:47.9062861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:47.9063524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:47.9064083Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:47.9077863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:47.9170643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:47.9172188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:48.0089353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:48.2883737Z ok (2.076s) 2022-08-17T13:29:48.2883997Z 2022-08-17T13:29:48.2884549Z ---------------------------------------------------------------------- 2022-08-17T13:29:48.2884847Z Ran 1 test in 2.077s 2022-08-17T13:29:48.2884971Z 2022-08-17T13:29:48.2885038Z OK 2022-08-17T13:29:48.2885135Z 2022-08-17T13:29:48.2885235Z Generating XML reports... 2022-08-17T13:29:48.2919439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132946.xml 2022-08-17T13:29:49.5994123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:49.5994538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:49.5997399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:49.5997787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:49.9417427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:49.9427195Z 2022-08-17T13:29:49.9427287Z Running tests... 2022-08-17T13:29:49.9428416Z ---------------------------------------------------------------------- 2022-08-17T13:29:50.2099684Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14819 2022-08-17T13:29:50.2126862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14820 2022-08-17T13:29:50.2155935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14821 2022-08-17T13:29:51.1969336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:51.1969926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:51.1972748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:51.1973329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:51.2473633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:51.2474167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:51.2477315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:51.2477889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:51.2481876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:51.2482384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:51.2486594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:51.2487136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:51.6109174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:51.6210186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:51.6210945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:51.6211606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:51.6212180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:51.6212892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:51.6318566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:51.6319182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:51.7222457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:52.2210178Z ok (2.278s) 2022-08-17T13:29:52.2210361Z 2022-08-17T13:29:52.2210892Z ---------------------------------------------------------------------- 2022-08-17T13:29:52.2211271Z Ran 1 test in 2.278s 2022-08-17T13:29:52.2211394Z 2022-08-17T13:29:52.2211456Z OK 2022-08-17T13:29:52.2211567Z 2022-08-17T13:29:52.2211659Z Generating XML reports... 2022-08-17T13:29:52.2245731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132949.xml 2022-08-17T13:29:53.5168396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:53.5169025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:53.5171364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:53.5171935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:53.8586117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:53.8596019Z 2022-08-17T13:29:53.8596340Z Running tests... 2022-08-17T13:29:53.8597012Z ---------------------------------------------------------------------- 2022-08-17T13:29:54.1243213Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14872 2022-08-17T13:29:54.1270590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14873 2022-08-17T13:29:54.1298304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14874 2022-08-17T13:29:55.1056446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:55.1057041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:55.1059538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:55.1060107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:55.1061390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:55.1062037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:55.1066089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:55.1066455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:55.1535860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:55.1536260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:55.1539869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:55.1540417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:55.5323636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:55.5425045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:55.5425828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:55.5426490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:55.5427228Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:55.5427769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:55.5433344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:55.5434263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:55.5435271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:55.5640692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:55.5741909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:55.5742541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:55.5743409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:55.5744015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:55.5744572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:56.0348847Z ok (2.175s) 2022-08-17T13:29:56.0349282Z 2022-08-17T13:29:56.0349779Z ---------------------------------------------------------------------- 2022-08-17T13:29:56.0350163Z Ran 1 test in 2.175s 2022-08-17T13:29:56.0350287Z 2022-08-17T13:29:56.0350355Z OK 2022-08-17T13:29:56.0350449Z 2022-08-17T13:29:56.0350532Z Generating XML reports... 2022-08-17T13:29:56.0385472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132953.xml 2022-08-17T13:29:57.3363287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:57.3363800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:57.3366187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:57.3366569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:57.6791871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:29:57.6801380Z 2022-08-17T13:29:57.6801516Z Running tests... 2022-08-17T13:29:57.6802209Z ---------------------------------------------------------------------- 2022-08-17T13:29:57.9470772Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14934 2022-08-17T13:29:57.9497544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14935 2022-08-17T13:29:57.9525539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14936 2022-08-17T13:29:59.0104213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:59.0104689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:59.0108041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:59.0108552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:59.0404492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:59.0404878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:59.0408429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:59.0409286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:59.0886590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:29:59.0886949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:29:59.0891450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:29:59.0891993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:29:59.4558233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:29:59.4658881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:29:59.4659620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:29:59.4660284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:59.4660851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:59.4661387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:29:59.4766357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:29:59.5672315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:29:59.5673048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:29:59.5673578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:29:59.5878992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:29:59.5879760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:29:59.5880544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:59.5881090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:29:59.5977978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:00.0578249Z ok (2.377s) 2022-08-17T13:30:00.0578543Z 2022-08-17T13:30:00.0579044Z ---------------------------------------------------------------------- 2022-08-17T13:30:00.0579539Z Ran 1 test in 2.378s 2022-08-17T13:30:00.0579661Z 2022-08-17T13:30:00.0579726Z OK 2022-08-17T13:30:00.0579809Z 2022-08-17T13:30:00.0579902Z Generating XML reports... 2022-08-17T13:30:00.0617123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132957.xml 2022-08-17T13:30:01.3564509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:01.3565231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:01.3568308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:01.3568899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:01.7091998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:01.7101605Z 2022-08-17T13:30:01.7101909Z Running tests... 2022-08-17T13:30:01.7102517Z ---------------------------------------------------------------------- 2022-08-17T13:30:01.7130139Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s) 2022-08-17T13:30:01.7130818Z 2022-08-17T13:30:01.7131053Z ---------------------------------------------------------------------- 2022-08-17T13:30:01.7131356Z Ran 1 test in 0.003s 2022-08-17T13:30:01.7131559Z 2022-08-17T13:30:01.7131647Z OK (skipped=1) 2022-08-17T13:30:01.7131747Z 2022-08-17T13:30:01.7131837Z Generating XML reports... 2022-08-17T13:30:01.7157358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133001.xml 2022-08-17T13:30:02.8964401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:02.8964781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:02.8967223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:02.8967596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:03.2360710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:03.2370483Z 2022-08-17T13:30:03.2370590Z Running tests... 2022-08-17T13:30:03.2371177Z ---------------------------------------------------------------------- 2022-08-17T13:30:03.5054506Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15003 2022-08-17T13:30:03.5082664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15004 2022-08-17T13:30:03.5111723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15005 2022-08-17T13:30:04.5483507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:04.5483919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:04.5486589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:04.5487166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:04.5538862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:04.5539348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:04.5543262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:04.5543806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:04.5569068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:04.5569730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:04.5573549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:04.5574076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:04.9164418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:04.9266286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:04.9267464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:04.9268243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:04.9269158Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:04.9269715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:04.9275497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:04.9276686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:04.9277632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:05.3158349Z ok (2.078s) 2022-08-17T13:30:05.3158591Z 2022-08-17T13:30:05.3159025Z ---------------------------------------------------------------------- 2022-08-17T13:30:05.3159446Z Ran 1 test in 2.079s 2022-08-17T13:30:05.3159653Z 2022-08-17T13:30:05.3159765Z OK 2022-08-17T13:30:05.3159917Z 2022-08-17T13:30:05.3160056Z Generating XML reports... 2022-08-17T13:30:05.3195988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133003.xml 2022-08-17T13:30:06.6271792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:06.6272198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:06.6274664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:06.6275038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:06.9674296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:06.9684194Z 2022-08-17T13:30:06.9684508Z Running tests... 2022-08-17T13:30:06.9685152Z ---------------------------------------------------------------------- 2022-08-17T13:30:07.2300132Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15056 2022-08-17T13:30:07.2329552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15057 2022-08-17T13:30:07.2357429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15058 2022-08-17T13:30:08.2219294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:08.2219964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:08.2222682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:08.2223301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:08.2505063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:08.2505495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:08.2508300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:08.2508959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:08.2553297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:08.2553676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:08.2557637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:08.2558014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:08.6229870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:08.6230668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:08.6231386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:08.6232298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:08.6232844Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:08.6233567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:08.6338301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:08.6338927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:08.7242636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:09.1404965Z ok (2.172s) 2022-08-17T13:30:09.1405125Z 2022-08-17T13:30:09.1405500Z ---------------------------------------------------------------------- 2022-08-17T13:30:09.1405776Z Ran 1 test in 2.172s 2022-08-17T13:30:09.1406099Z 2022-08-17T13:30:09.1406187Z OK 2022-08-17T13:30:09.1406285Z 2022-08-17T13:30:09.1406381Z Generating XML reports... 2022-08-17T13:30:09.1440788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133006.xml 2022-08-17T13:30:10.4244964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:10.4245370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:10.4248013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:10.4248442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:10.7723565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:10.7733719Z 2022-08-17T13:30:10.7733824Z Running tests... 2022-08-17T13:30:10.7734335Z ---------------------------------------------------------------------- 2022-08-17T13:30:11.0424917Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15109 2022-08-17T13:30:11.0452929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15110 2022-08-17T13:30:11.0483121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15111 2022-08-17T13:30:12.0309232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:12.0309929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:12.0312211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:12.0312795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:12.0403515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:12.0404197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:12.0408105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:12.0408656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:12.0788938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:12.0789327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:12.0792028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:12.0792529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:12.4376408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:12.4477512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:12.4478102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:12.4478965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:12.4479539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:12.4485645Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:12.4486326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:12.4487710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:12.4488736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:12.4489605Z STAGE:2022-08-17 13:30:12 15109:15109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:12.4491107Z STAGE:2022-08-17 13:30:12 15110:15110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:12.4491664Z STAGE:2022-08-17 13:30:12 15111:15111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:12.4525361Z STAGE:2022-08-17 13:30:12 15110:15110 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:12.4526422Z STAGE:2022-08-17 13:30:12 15111:15111 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:12.5512546Z STAGE:2022-08-17 13:30:12 15109:15109 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:12.8532103Z ok (2.079s) 2022-08-17T13:30:12.8532363Z 2022-08-17T13:30:12.8532907Z ---------------------------------------------------------------------- 2022-08-17T13:30:12.8533323Z Ran 1 test in 2.080s 2022-08-17T13:30:12.8533444Z 2022-08-17T13:30:12.8533494Z OK 2022-08-17T13:30:12.8533654Z 2022-08-17T13:30:12.8533753Z Generating XML reports... 2022-08-17T13:30:12.8568149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133010.xml 2022-08-17T13:30:14.1541253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:14.1541654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:14.1544695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:14.1545099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:14.4984880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:14.4994156Z 2022-08-17T13:30:14.4994596Z Running tests... 2022-08-17T13:30:14.4994959Z ---------------------------------------------------------------------- 2022-08-17T13:30:14.7648427Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15165 2022-08-17T13:30:14.7675867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15166 2022-08-17T13:30:14.7704135Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15167 2022-08-17T13:30:15.7621672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:15.7622360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:15.7653703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:15.7654603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:15.7755674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:15.7756338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:15.7760125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:15.7760554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:15.7769080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:15.7769539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:15.7773196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:15.7773710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:16.1450618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:16.1551581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:16.1552267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:16.1553041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:16.1553594Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:16.1554143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:16.1659256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:16.1662102Z STAGE:2022-08-17 13:30:16 15167:15167 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:16.2564736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:16.2565184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:16.2567854Z STAGE:2022-08-17 13:30:16 15165:15165 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:16.2568494Z STAGE:2022-08-17 13:30:16 15166:15166 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:30:16.2601788Z STAGE:2022-08-17 13:30:16 15165:15165 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:16.2603226Z STAGE:2022-08-17 13:30:16 15166:15166 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:16.2604137Z STAGE:2022-08-17 13:30:16 15167:15167 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:30:16.6753508Z ok (2.176s) 2022-08-17T13:30:16.6753752Z 2022-08-17T13:30:16.6754303Z ---------------------------------------------------------------------- 2022-08-17T13:30:16.6754855Z Ran 1 test in 2.176s 2022-08-17T13:30:16.6754979Z 2022-08-17T13:30:16.6755030Z OK 2022-08-17T13:30:16.6755125Z 2022-08-17T13:30:16.6755224Z Generating XML reports... 2022-08-17T13:30:16.6789572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133014.xml 2022-08-17T13:30:17.9888644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:17.9889050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:17.9891245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:17.9891623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:18.3312981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:18.3322770Z 2022-08-17T13:30:18.3322873Z Running tests... 2022-08-17T13:30:18.3323485Z ---------------------------------------------------------------------- 2022-08-17T13:30:18.3342437Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:30:18.3342833Z 2022-08-17T13:30:18.3343277Z ---------------------------------------------------------------------- 2022-08-17T13:30:18.3343547Z Ran 1 test in 0.002s 2022-08-17T13:30:18.3343669Z 2022-08-17T13:30:18.3343747Z OK (skipped=1) 2022-08-17T13:30:18.3343863Z 2022-08-17T13:30:18.3343955Z Generating XML reports... 2022-08-17T13:30:18.3367838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133018.xml 2022-08-17T13:30:19.4796262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:19.4796767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:19.4798509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:19.4798892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:19.8203245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:19.8213474Z 2022-08-17T13:30:19.8213577Z Running tests... 2022-08-17T13:30:19.8213980Z ---------------------------------------------------------------------- 2022-08-17T13:30:19.8231826Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:30:19.8232338Z 2022-08-17T13:30:19.8232739Z ---------------------------------------------------------------------- 2022-08-17T13:30:19.8233053Z Ran 1 test in 0.002s 2022-08-17T13:30:19.8233237Z 2022-08-17T13:30:19.8233301Z OK (skipped=1) 2022-08-17T13:30:19.8233427Z 2022-08-17T13:30:19.8233516Z Generating XML reports... 2022-08-17T13:30:19.8257806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133019.xml 2022-08-17T13:30:20.9767816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:20.9768273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:20.9770922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:20.9771331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:21.3196816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:21.3206733Z 2022-08-17T13:30:21.3206980Z Running tests... 2022-08-17T13:30:21.3207614Z ---------------------------------------------------------------------- 2022-08-17T13:30:21.5836243Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15241 2022-08-17T13:30:21.5864801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15242 2022-08-17T13:30:21.5893794Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15243 2022-08-17T13:30:22.5723448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:22.5723841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:22.5726895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:22.5727391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:22.5874057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:22.5874561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:22.5878939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:22.5879913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:22.6458693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:22.6459433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:22.6461203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:22.6461938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:23.0162650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:23.0264306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:23.0264748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:23.0265620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:23.0266555Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:23.0267172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:23.0272759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:23.0273387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:23.0274903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:25.0302433Z [E ProcessGroupGloo.cpp:136] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:30:25.0401574Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:30:25.4973168Z ok (4.176s) 2022-08-17T13:30:25.4973434Z 2022-08-17T13:30:25.4973811Z ---------------------------------------------------------------------- 2022-08-17T13:30:25.4974094Z Ran 1 test in 4.177s 2022-08-17T13:30:25.4974214Z 2022-08-17T13:30:25.4974277Z OK 2022-08-17T13:30:25.4974372Z 2022-08-17T13:30:25.4974467Z Generating XML reports... 2022-08-17T13:30:25.5009380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133021.xml 2022-08-17T13:30:26.8161113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:26.8162067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:26.8164481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:26.8165057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:27.1584296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:27.1594027Z 2022-08-17T13:30:27.1594133Z Running tests... 2022-08-17T13:30:27.1594618Z ---------------------------------------------------------------------- 2022-08-17T13:30:27.4287340Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15294 2022-08-17T13:30:27.4314925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15295 2022-08-17T13:30:27.4343329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15296 2022-08-17T13:30:28.4197130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:28.4197765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:28.4200891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:28.4201557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:28.4345420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:28.4346115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:28.4350259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:28.4350727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:28.4418983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:28.4419373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:28.4423779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:28.4424958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:28.8110266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:28.8110744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:28.8111137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:28.8111760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:28.8112305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:28.8112871Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:28.8217386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:28.9122084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:28.9122481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:30.9331428Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:30:30.9332016Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:30:31.3422242Z ok (4.182s) 2022-08-17T13:30:31.3422567Z 2022-08-17T13:30:31.3423236Z ---------------------------------------------------------------------- 2022-08-17T13:30:31.3423529Z Ran 1 test in 4.183s 2022-08-17T13:30:31.3423652Z 2022-08-17T13:30:31.3423715Z OK 2022-08-17T13:30:31.3423809Z 2022-08-17T13:30:31.3423904Z Generating XML reports... 2022-08-17T13:30:31.3457891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133027.xml 2022-08-17T13:30:32.6344497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:32.6344907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:32.6347408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:32.6347780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:32.9839076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:32.9850256Z 2022-08-17T13:30:32.9850518Z Running tests... 2022-08-17T13:30:33.2543277Z ---------------------------------------------------------------------- 2022-08-17T13:30:33.2544481Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15347 2022-08-17T13:30:33.2571808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15348 2022-08-17T13:30:33.2600384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15349 2022-08-17T13:30:34.2547419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:34.2548020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:34.2550650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:34.2551303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:34.2659522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:34.2660191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:34.2664065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:34.2664739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:34.2815751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:34.2816368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:34.2818005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:34.2818634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:34.6558373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:34.6559065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:34.6559457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:34.6560110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:34.6560669Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:34.6561204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:34.6663956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:34.7571594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:34.7572430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:34.7879913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:30:34.7981818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:30:34.7982371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:30:34.7983121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:34.7983705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:34.7984336Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:34.7985425Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-08-17T13:30:34.7985777Z No ranks successfully processed in monitoredBarrier. 2022-08-17T13:30:34.8013818Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-08-17T13:30:35.2653056Z ok (2.280s) 2022-08-17T13:30:35.2653359Z 2022-08-17T13:30:35.2653888Z ---------------------------------------------------------------------- 2022-08-17T13:30:35.2654352Z Ran 1 test in 2.280s 2022-08-17T13:30:35.2654573Z 2022-08-17T13:30:35.2654685Z OK 2022-08-17T13:30:35.2654841Z 2022-08-17T13:30:35.2655007Z Generating XML reports... 2022-08-17T13:30:35.2690035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133032.xml 2022-08-17T13:30:36.5765593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:36.5766186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:36.5768271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:36.5768648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:36.9192687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:36.9203503Z 2022-08-17T13:30:36.9203792Z Running tests... 2022-08-17T13:30:36.9204444Z ---------------------------------------------------------------------- 2022-08-17T13:30:37.1851502Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15409 2022-08-17T13:30:37.1879456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15410 2022-08-17T13:30:37.1908195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15411 2022-08-17T13:30:38.1737447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:38.1738070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:38.1745440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:38.1745926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:38.1891191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:38.1894653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:38.1895265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:38.1895626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:38.2038301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:38.2038911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:38.2042190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:38.2042769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:38.5758096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:38.5859332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:38.5859834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:38.5860489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:38.5861050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:38.5861621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:38.5966712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:38.5968651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:30:38.6872301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:38.6872891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:38.7079679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:30:38.7080355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:30:38.7081241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:38.7081973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:38.7082611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:30:38.7089405Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:299: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group. 2022-08-17T13:30:38.7089857Z f"Running {op_name} on global rank {global_rank} which does not " 2022-08-17T13:30:38.8185700Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:30:39.1959924Z ok (2.275s) 2022-08-17T13:30:39.1960182Z 2022-08-17T13:30:39.1960583Z ---------------------------------------------------------------------- 2022-08-17T13:30:39.1960869Z Ran 1 test in 2.276s 2022-08-17T13:30:39.1961005Z 2022-08-17T13:30:39.1961072Z OK 2022-08-17T13:30:39.1961171Z 2022-08-17T13:30:39.1961257Z Generating XML reports... 2022-08-17T13:30:39.1999752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133036.xml 2022-08-17T13:30:40.4966764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:40.4967205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:40.4969815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:40.4970298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:40.8468956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:40.8478981Z 2022-08-17T13:30:40.8479460Z Running tests... 2022-08-17T13:30:40.8480143Z ---------------------------------------------------------------------- 2022-08-17T13:30:41.1143800Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15468 2022-08-17T13:30:41.1171819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15469 2022-08-17T13:30:41.1199772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15470 2022-08-17T13:30:42.0762975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:42.0763422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:42.0765715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:42.0766310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:42.1191524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:42.1192113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:42.1195867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:42.1196255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:42.1384060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:42.1384470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:42.1388173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:42.1388774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:42.4992295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:42.4992757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:42.4993127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:42.4993770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:42.4994322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:42.4994868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:42.5001102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:42.5001856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:42.5002838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:42.6197276Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:30:42.6197676Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:30:42.6198069Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:30:43.0253036Z ok (2.177s) 2022-08-17T13:30:43.0253336Z 2022-08-17T13:30:43.0253882Z ---------------------------------------------------------------------- 2022-08-17T13:30:43.0254213Z Ran 1 test in 2.177s 2022-08-17T13:30:43.0254337Z 2022-08-17T13:30:43.0254403Z OK 2022-08-17T13:30:43.0254486Z 2022-08-17T13:30:43.0254578Z Generating XML reports... 2022-08-17T13:30:43.0288938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133040.xml 2022-08-17T13:30:44.3002390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:44.3002772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:44.3005502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:44.3006052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:44.6362422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:44.6372540Z 2022-08-17T13:30:44.6372657Z Running tests... 2022-08-17T13:30:44.6373482Z ---------------------------------------------------------------------- 2022-08-17T13:30:44.6398638Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:30:44.6399067Z 2022-08-17T13:30:44.6399349Z ---------------------------------------------------------------------- 2022-08-17T13:30:44.6399702Z Ran 1 test in 0.003s 2022-08-17T13:30:44.6399859Z 2022-08-17T13:30:44.6399951Z OK (skipped=1) 2022-08-17T13:30:44.6400063Z 2022-08-17T13:30:44.6400154Z Generating XML reports... 2022-08-17T13:30:44.6424616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133044.xml 2022-08-17T13:30:45.7785680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:45.7786078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:45.7788806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:45.7789182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:46.1195284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:46.1204828Z 2022-08-17T13:30:46.1204927Z Running tests... 2022-08-17T13:30:46.1206560Z ---------------------------------------------------------------------- 2022-08-17T13:30:46.1230660Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:30:46.1231125Z 2022-08-17T13:30:46.1231526Z ---------------------------------------------------------------------- 2022-08-17T13:30:46.1231830Z Ran 1 test in 0.003s 2022-08-17T13:30:46.1231949Z 2022-08-17T13:30:46.1232027Z OK (skipped=1) 2022-08-17T13:30:46.1232145Z 2022-08-17T13:30:46.1232235Z Generating XML reports... 2022-08-17T13:30:46.1256177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133046.xml 2022-08-17T13:30:47.2619753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:47.2620143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:47.2622384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:47.2622983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:47.6000959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:47.6010728Z 2022-08-17T13:30:47.6010877Z Running tests... 2022-08-17T13:30:47.6011512Z ---------------------------------------------------------------------- 2022-08-17T13:30:47.6035139Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:30:47.6035562Z 2022-08-17T13:30:47.6035828Z ---------------------------------------------------------------------- 2022-08-17T13:30:47.6036149Z Ran 1 test in 0.002s 2022-08-17T13:30:47.6036273Z 2022-08-17T13:30:47.6036352Z OK (skipped=1) 2022-08-17T13:30:47.6036464Z 2022-08-17T13:30:47.6036782Z Generating XML reports... 2022-08-17T13:30:47.6061605Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133047.xml 2022-08-17T13:30:48.7490706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:48.7491139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:48.7493441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:48.7493931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:49.0945348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:49.0954735Z 2022-08-17T13:30:49.0954884Z Running tests... 2022-08-17T13:30:49.0955519Z ---------------------------------------------------------------------- 2022-08-17T13:30:49.0984722Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2022-08-17T13:30:49.0985190Z 2022-08-17T13:30:49.0985537Z ---------------------------------------------------------------------- 2022-08-17T13:30:49.0986005Z Ran 1 test in 0.003s 2022-08-17T13:30:49.0986203Z 2022-08-17T13:30:49.0986504Z OK (skipped=1) 2022-08-17T13:30:49.0986627Z 2022-08-17T13:30:49.0986716Z Generating XML reports... 2022-08-17T13:30:49.1010734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133049.xml 2022-08-17T13:30:50.2522299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:50.2522682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:50.2525305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:50.2526087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:50.5979702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:50.5989844Z 2022-08-17T13:30:50.5990129Z Running tests... 2022-08-17T13:30:50.5990824Z ---------------------------------------------------------------------- 2022-08-17T13:30:50.6012807Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-08-17T13:30:50.6013286Z 2022-08-17T13:30:50.6013706Z ---------------------------------------------------------------------- 2022-08-17T13:30:50.6014114Z Ran 1 test in 0.002s 2022-08-17T13:30:50.6014309Z 2022-08-17T13:30:50.6014435Z OK (skipped=1) 2022-08-17T13:30:50.6014633Z 2022-08-17T13:30:50.6014789Z Generating XML reports... 2022-08-17T13:30:50.6040240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133050.xml 2022-08-17T13:30:51.7772955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:51.7773383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:51.7775825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:51.7776225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:52.1219006Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:52.1228819Z 2022-08-17T13:30:52.1228998Z Running tests... 2022-08-17T13:30:52.1229479Z ---------------------------------------------------------------------- 2022-08-17T13:30:52.1249899Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:30:52.1250206Z 2022-08-17T13:30:52.1250604Z ---------------------------------------------------------------------- 2022-08-17T13:30:52.1251118Z Ran 1 test in 0.002s 2022-08-17T13:30:52.1251286Z 2022-08-17T13:30:52.1251382Z OK (skipped=1) 2022-08-17T13:30:52.1251499Z 2022-08-17T13:30:52.1251594Z Generating XML reports... 2022-08-17T13:30:52.1276033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133052.xml 2022-08-17T13:30:53.3049522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:53.3049988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:53.3052970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:53.3053440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:53.6515885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:53.6526366Z 2022-08-17T13:30:53.6526833Z Running tests... 2022-08-17T13:30:53.6527345Z ---------------------------------------------------------------------- 2022-08-17T13:30:53.6549945Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:30:53.6550411Z 2022-08-17T13:30:53.6551176Z ---------------------------------------------------------------------- 2022-08-17T13:30:53.6551670Z Ran 1 test in 0.002s 2022-08-17T13:30:53.6551891Z 2022-08-17T13:30:53.6552023Z OK (skipped=1) 2022-08-17T13:30:53.6552168Z 2022-08-17T13:30:53.6552261Z Generating XML reports... 2022-08-17T13:30:53.6576178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133053.xml 2022-08-17T13:30:54.8033004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:54.8033382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:54.8036061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:54.8036482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:55.1428200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:55.1437757Z 2022-08-17T13:30:55.1437918Z Running tests... 2022-08-17T13:30:55.1438369Z ---------------------------------------------------------------------- 2022-08-17T13:30:55.1458403Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:30:55.1458906Z 2022-08-17T13:30:55.1459558Z ---------------------------------------------------------------------- 2022-08-17T13:30:55.1459958Z Ran 1 test in 0.002s 2022-08-17T13:30:55.1460089Z 2022-08-17T13:30:55.1460183Z OK (skipped=1) 2022-08-17T13:30:55.1460283Z 2022-08-17T13:30:55.1460387Z Generating XML reports... 2022-08-17T13:30:55.1484625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133055.xml 2022-08-17T13:30:56.2842890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:56.2843304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:56.2846830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:56.2847236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:56.6268265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:30:56.6278142Z 2022-08-17T13:30:56.6278236Z Running tests... 2022-08-17T13:30:56.6279091Z ---------------------------------------------------------------------- 2022-08-17T13:30:56.8905561Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15601 2022-08-17T13:30:56.8933378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15602 2022-08-17T13:30:56.8963197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15603 2022-08-17T13:30:57.9035988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:57.9039661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:57.9040210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:57.9040596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:57.9706349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:57.9706748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:57.9709893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:57.9710601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:57.9711791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:30:57.9712163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:30:57.9715971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:30:57.9716521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:30:58.3401409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:30:58.3446759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:30:58.3447550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:30:58.3448291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:58.3448846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:58.3503445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:30:58.3555963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:30:58.4515481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:30:58.4558929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:30:58.8014331Z skip: CUDA is not available. (2.173s) 2022-08-17T13:30:58.8014541Z 2022-08-17T13:30:58.8015104Z ---------------------------------------------------------------------- 2022-08-17T13:30:58.8015491Z Ran 1 test in 2.174s 2022-08-17T13:30:58.8015610Z 2022-08-17T13:30:58.8015685Z OK (skipped=1) 2022-08-17T13:30:58.8015796Z 2022-08-17T13:30:58.8015902Z Generating XML reports... 2022-08-17T13:30:58.8053373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133056.xml 2022-08-17T13:31:00.0971228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:00.0971625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:00.0972943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:00.0973464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:00.4399481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:00.4408918Z 2022-08-17T13:31:00.4409102Z Running tests... 2022-08-17T13:31:00.4409580Z ---------------------------------------------------------------------- 2022-08-17T13:31:00.7061189Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15654 2022-08-17T13:31:00.7089148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15655 2022-08-17T13:31:00.7117451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15656 2022-08-17T13:31:01.7010036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:01.7010646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:01.7013926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:01.7014591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:01.7044760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:01.7045656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:01.7049212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:01.7049623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:01.7069704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:01.7070136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:01.7073800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:01.7074286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:02.0704125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:02.0804530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:02.0805313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:02.0806162Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:02.0806716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:02.0807271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:02.0911849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:02.1818648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:02.1819212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:02.5165673Z skip: CUDA is not available. (2.075s) 2022-08-17T13:31:02.5166164Z 2022-08-17T13:31:02.5166646Z ---------------------------------------------------------------------- 2022-08-17T13:31:02.5166919Z Ran 1 test in 2.076s 2022-08-17T13:31:02.5167028Z 2022-08-17T13:31:02.5167107Z OK (skipped=1) 2022-08-17T13:31:02.5168473Z 2022-08-17T13:31:02.5168577Z Generating XML reports... 2022-08-17T13:31:02.5201684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133100.xml 2022-08-17T13:31:03.7943258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:03.7943917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:03.7945790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:03.7946173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:04.1345762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:04.1355194Z 2022-08-17T13:31:04.1355433Z Running tests... 2022-08-17T13:31:04.1356032Z ---------------------------------------------------------------------- 2022-08-17T13:31:04.1374770Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:31:04.1375188Z 2022-08-17T13:31:04.1375462Z ---------------------------------------------------------------------- 2022-08-17T13:31:04.1375722Z Ran 1 test in 0.002s 2022-08-17T13:31:04.1375847Z 2022-08-17T13:31:04.1375938Z OK (skipped=1) 2022-08-17T13:31:04.1376053Z 2022-08-17T13:31:04.1376141Z Generating XML reports... 2022-08-17T13:31:04.1400761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133104.xml 2022-08-17T13:31:05.2895745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:05.2896275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:05.2899017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:05.2899477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:05.6336247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:05.6347210Z 2022-08-17T13:31:05.6347496Z Running tests... 2022-08-17T13:31:05.6348143Z ---------------------------------------------------------------------- 2022-08-17T13:31:05.6366486Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:31:05.6366942Z 2022-08-17T13:31:05.6367285Z ---------------------------------------------------------------------- 2022-08-17T13:31:05.6367623Z Ran 1 test in 0.002s 2022-08-17T13:31:05.6367751Z 2022-08-17T13:31:05.6367828Z OK (skipped=1) 2022-08-17T13:31:05.6367938Z 2022-08-17T13:31:05.6368031Z Generating XML reports... 2022-08-17T13:31:05.6393150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133105.xml 2022-08-17T13:31:06.8057580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:06.8058282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:06.8061469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:06.8062069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:07.1534571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:07.1545366Z 2022-08-17T13:31:07.1545849Z Running tests... 2022-08-17T13:31:07.1546302Z ---------------------------------------------------------------------- 2022-08-17T13:31:07.4117829Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s) 2022-08-17T13:31:07.4118655Z 2022-08-17T13:31:07.4118981Z ---------------------------------------------------------------------- 2022-08-17T13:31:07.4119651Z Ran 1 test in 0.257s 2022-08-17T13:31:07.4119811Z 2022-08-17T13:31:07.4119910Z OK (skipped=1) 2022-08-17T13:31:07.4120060Z 2022-08-17T13:31:07.4120170Z Generating XML reports... 2022-08-17T13:31:07.4145321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133107.xml 2022-08-17T13:31:08.6764389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:08.6765013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:08.6767617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:08.6768199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:09.0250697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:09.0261142Z 2022-08-17T13:31:09.0261524Z Running tests... 2022-08-17T13:31:09.0262227Z ---------------------------------------------------------------------- 2022-08-17T13:31:09.2929133Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15737 2022-08-17T13:31:09.2957647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15738 2022-08-17T13:31:09.2986083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15739 2022-08-17T13:31:10.2647724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:10.2648368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:10.2649105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:10.2649677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:10.2650815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:10.2651428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:10.2653628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:10.2654229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:10.2886012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:10.2886437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:10.2890257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:10.2890640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:10.6752230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:10.6752992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:10.6753368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:10.6754012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:10.6754574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:10.6755127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:10.6760964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:10.6761570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:10.6762153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:11.0032579Z skip: Need at least 2 CUDA devices (1.977s) 2022-08-17T13:31:11.0032872Z 2022-08-17T13:31:11.0033196Z ---------------------------------------------------------------------- 2022-08-17T13:31:11.0033533Z Ran 1 test in 1.977s 2022-08-17T13:31:11.0033673Z 2022-08-17T13:31:11.0033738Z OK (skipped=1) 2022-08-17T13:31:11.0033855Z 2022-08-17T13:31:11.0033945Z Generating XML reports... 2022-08-17T13:31:11.0067732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133109.xml 2022-08-17T13:31:12.2780319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:12.2780962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:12.2782489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:12.2783134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:12.6253776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:12.6263847Z 2022-08-17T13:31:12.6264471Z Running tests... 2022-08-17T13:31:12.6265251Z ---------------------------------------------------------------------- 2022-08-17T13:31:12.8892779Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15790 2022-08-17T13:31:12.8919460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15791 2022-08-17T13:31:12.8946781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15792 2022-08-17T13:31:13.8911350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:13.8911987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:13.8914952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:13.8915432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:13.9014960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:13.9015648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:13.9018545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:13.9019197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:13.9221210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:13.9221873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:13.9224431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:13.9225095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:14.2949857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:14.3051425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:14.3052094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:14.3053080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:14.3053744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:14.3054291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:14.3060426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:14.3061986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:14.3063113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:14.5993300Z skip: Need at least 2 CUDA devices (1.973s) 2022-08-17T13:31:14.5993604Z 2022-08-17T13:31:14.6005680Z ---------------------------------------------------------------------- 2022-08-17T13:31:14.6006429Z Ran 1 test in 1.973s 2022-08-17T13:31:14.6006634Z 2022-08-17T13:31:14.6006764Z OK (skipped=1) 2022-08-17T13:31:14.6006961Z 2022-08-17T13:31:14.6007112Z Generating XML reports... 2022-08-17T13:31:14.6034083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133112.xml 2022-08-17T13:31:15.8847067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:15.8847456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:15.8850107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:15.8850634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:16.2322565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:16.2332689Z 2022-08-17T13:31:16.2332828Z Running tests... 2022-08-17T13:31:16.2333571Z ---------------------------------------------------------------------- 2022-08-17T13:31:16.5021225Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15843 2022-08-17T13:31:16.5049801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15844 2022-08-17T13:31:16.5078525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15845 2022-08-17T13:31:17.4962470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:17.4963181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:17.4965148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:17.4965676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:17.4988309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:17.4988873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:17.4992700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:17.4993267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:17.5202287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:17.5202747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:17.5206214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:17.5206722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:17.9011137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:17.9011932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:17.9012648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:17.9013580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:17.9014352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:17.9014912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:17.9021355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:17.9021728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:18.0025407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:18.3126156Z skip: Need at least 2 CUDA devices (2.079s) 2022-08-17T13:31:18.3126494Z 2022-08-17T13:31:18.3126967Z ---------------------------------------------------------------------- 2022-08-17T13:31:18.3127235Z Ran 1 test in 2.079s 2022-08-17T13:31:18.3127370Z 2022-08-17T13:31:18.3127448Z OK (skipped=1) 2022-08-17T13:31:18.3127563Z 2022-08-17T13:31:18.3127652Z Generating XML reports... 2022-08-17T13:31:18.3162647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133116.xml 2022-08-17T13:31:19.6145638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:19.6146259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:19.6148325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:19.6148957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:19.9577886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:19.9589345Z 2022-08-17T13:31:19.9589700Z Running tests... 2022-08-17T13:31:19.9590376Z ---------------------------------------------------------------------- 2022-08-17T13:31:20.2205254Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15896 2022-08-17T13:31:20.2234372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15897 2022-08-17T13:31:20.2262857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15898 2022-08-17T13:31:21.2211795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:21.2212450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:21.2213066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:21.2213486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:21.2215219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:21.2215922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:21.2216747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:21.2217307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:21.2294836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:21.2298753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:21.2299632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:21.2300271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:21.5889188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:21.5989773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:21.5990340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:21.5990987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:21.5991550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:21.5992102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:21.6097682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:21.6098093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:21.7003412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:22.0308975Z skip: Need at least 2 CUDA devices (2.072s) 2022-08-17T13:31:22.0309243Z 2022-08-17T13:31:22.0309757Z ---------------------------------------------------------------------- 2022-08-17T13:31:22.0310228Z Ran 1 test in 2.072s 2022-08-17T13:31:22.0310445Z 2022-08-17T13:31:22.0310867Z OK (skipped=1) 2022-08-17T13:31:22.0311073Z 2022-08-17T13:31:22.0311205Z Generating XML reports... 2022-08-17T13:31:22.0345662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133119.xml 2022-08-17T13:31:23.3243445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:23.3244192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:23.3247446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:23.3248095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:23.6713174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:23.6722951Z 2022-08-17T13:31:23.6723356Z Running tests... 2022-08-17T13:31:23.6723806Z ---------------------------------------------------------------------- 2022-08-17T13:31:23.9422916Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15949 2022-08-17T13:31:23.9452696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15950 2022-08-17T13:31:23.9481263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15951 2022-08-17T13:31:24.9370666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:24.9371350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:24.9373530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:24.9374179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:24.9629315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:24.9629721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:24.9632600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:24.9633293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:24.9978433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:24.9979061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:24.9982137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:24.9982874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:25.3617292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:25.3719433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:25.3720031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:25.3720685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:25.3721259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:25.3721812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:25.3728937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:25.3729932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:25.3730948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:25.6530180Z skip: Need at least 2 CUDA devices (1.980s) 2022-08-17T13:31:25.6530491Z 2022-08-17T13:31:25.6531023Z ---------------------------------------------------------------------- 2022-08-17T13:31:25.6531337Z Ran 1 test in 1.981s 2022-08-17T13:31:25.6531459Z 2022-08-17T13:31:25.6531522Z OK (skipped=1) 2022-08-17T13:31:25.6531634Z 2022-08-17T13:31:25.6531728Z Generating XML reports... 2022-08-17T13:31:25.6566714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133123.xml 2022-08-17T13:31:26.9528410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:26.9529127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:26.9532397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:26.9533023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:27.3011596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:27.3021348Z 2022-08-17T13:31:27.3021689Z Running tests... 2022-08-17T13:31:27.3022149Z ---------------------------------------------------------------------- 2022-08-17T13:31:27.5674021Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16002 2022-08-17T13:31:27.5703200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16003 2022-08-17T13:31:27.5731573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16004 2022-08-17T13:31:28.5802994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:28.5803670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:28.5807056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:28.5807716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:28.6048185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:28.6048841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:28.6050072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:28.6050937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:28.6051877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:28.6052456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:28.6056927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:28.6057571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:28.9814777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:28.9815476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:28.9816007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:28.9816684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:28.9817278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:28.9818427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:28.9822416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:28.9823132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:28.9824368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:29.2781633Z skip: Need at least 4 CUDA devices (1.976s) 2022-08-17T13:31:29.2781966Z 2022-08-17T13:31:29.2782608Z ---------------------------------------------------------------------- 2022-08-17T13:31:29.2782989Z Ran 1 test in 1.976s 2022-08-17T13:31:29.2783123Z 2022-08-17T13:31:29.2783200Z OK (skipped=1) 2022-08-17T13:31:29.2783301Z 2022-08-17T13:31:29.2783389Z Generating XML reports... 2022-08-17T13:31:29.2820823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133127.xml 2022-08-17T13:31:30.5835493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:30.5835907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:30.5838312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:30.5838763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:30.9349993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:30.9359973Z 2022-08-17T13:31:30.9360089Z Running tests... 2022-08-17T13:31:30.9360495Z ---------------------------------------------------------------------- 2022-08-17T13:31:31.2039273Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16055 2022-08-17T13:31:31.2068794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16056 2022-08-17T13:31:31.2106501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16057 2022-08-17T13:31:32.1749718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:32.1750335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:32.1751606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:32.1752217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:32.1762184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:32.1763101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:32.1767058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:32.1767665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:32.1826715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:32.1827371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:32.1831532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:32.1832170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:32.5535173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:32.5636153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:32.5636577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:32.5637420Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:32.5637990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:32.5638525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:32.5743092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:32.6649602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:32.6650209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:33.0155513Z skip: Need at least 4 CUDA devices (2.079s) 2022-08-17T13:31:33.0155800Z 2022-08-17T13:31:33.0156359Z ---------------------------------------------------------------------- 2022-08-17T13:31:33.0156658Z Ran 1 test in 2.079s 2022-08-17T13:31:33.0156795Z 2022-08-17T13:31:33.0156875Z OK (skipped=1) 2022-08-17T13:31:33.0156988Z 2022-08-17T13:31:33.0157065Z Generating XML reports... 2022-08-17T13:31:33.0195424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133130.xml 2022-08-17T13:31:34.3156292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:34.3156987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:34.3160051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:34.3160659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:34.6555227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:34.6566085Z 2022-08-17T13:31:34.6566419Z Running tests... 2022-08-17T13:31:34.6567123Z ---------------------------------------------------------------------- 2022-08-17T13:31:34.9189877Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16108 2022-08-17T13:31:34.9218012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16109 2022-08-17T13:31:34.9246431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16110 2022-08-17T13:31:35.9158839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:35.9159482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:35.9162015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:35.9162616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:35.9191133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:35.9191795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:35.9195002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:35.9195676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:35.9403242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:35.9403903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:35.9407346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:35.9408005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:36.3172356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:36.3273429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:36.3274185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:36.3274844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:36.3275413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:36.3275966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:36.3281966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:36.3282501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:36.3283541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:36.6292379Z skip: Need at least 2 CUDA devices (1.972s) 2022-08-17T13:31:36.6292695Z 2022-08-17T13:31:36.6293239Z ---------------------------------------------------------------------- 2022-08-17T13:31:36.6293582Z Ran 1 test in 1.973s 2022-08-17T13:31:36.6293689Z 2022-08-17T13:31:36.6293766Z OK (skipped=1) 2022-08-17T13:31:36.6293878Z 2022-08-17T13:31:36.6293969Z Generating XML reports... 2022-08-17T13:31:36.6327961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133134.xml 2022-08-17T13:31:37.9153030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:37.9153718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:37.9156229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:37.9156821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:38.2569455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:38.2580043Z 2022-08-17T13:31:38.2580416Z Running tests... 2022-08-17T13:31:38.2580866Z ---------------------------------------------------------------------- 2022-08-17T13:31:38.5262557Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16161 2022-08-17T13:31:38.5291141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16162 2022-08-17T13:31:38.5319919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16163 2022-08-17T13:31:39.4930418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:39.4930818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:39.4934158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:39.4934842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:39.4975025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:39.4975416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:39.4978681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:39.4979087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:39.5042998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:39.5043669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:39.5048495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:39.5049173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:39.8705449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:39.8806577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:39.8807204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:39.8807950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:39.8808836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:39.8809688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:39.8913487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:39.9818999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:39.9819556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:40.0128082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:40.0128815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:40.0129402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:40.0130300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:40.0131123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:40.0132009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:40.0132764Z STAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0133306Z STAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0133888Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0137655Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0138337Z STAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0138777Z STAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0147452Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0150453Z STAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0155965Z STAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0161641Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0162714Z STAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0163227Z 2022-08-17T13:31:40.0167496Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0169332Z STAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0174341Z STAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:40.0179998Z STAGE:2022-08-17 13:31:40 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0180867Z STAGE:2022-08-17 13:31:40 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:31:40 16161:16161 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:40.0181179Z 2022-08-17T13:31:40.4370440Z ok (2.179s) 2022-08-17T13:31:40.4370665Z 2022-08-17T13:31:40.4370995Z ---------------------------------------------------------------------- 2022-08-17T13:31:40.4371321Z Ran 1 test in 2.179s 2022-08-17T13:31:40.4371443Z 2022-08-17T13:31:40.4371507Z OK 2022-08-17T13:31:40.4371605Z 2022-08-17T13:31:40.4371699Z Generating XML reports... 2022-08-17T13:31:40.4406563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133138.xml 2022-08-17T13:31:41.7570846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:41.7571245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:41.7574470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:41.7574870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:42.1057520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:42.1067291Z 2022-08-17T13:31:42.1067457Z Running tests... 2022-08-17T13:31:42.1068144Z ---------------------------------------------------------------------- 2022-08-17T13:31:42.3713498Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16226 2022-08-17T13:31:42.3741543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16227 2022-08-17T13:31:42.3769696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16228 2022-08-17T13:31:43.3550596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:43.3551277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:43.3554431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:43.3555061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:43.3775162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:43.3776140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:43.3779091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:43.3779763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:43.3785496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:43.3786147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:43.3790047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:43.3790502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:43.7423108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:43.7513247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:43.7513899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:43.7514727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:43.7515303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:43.7523880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:43.7620745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:43.7621404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:43.8533863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:43.8732915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:43.8835128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:43.8835916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:43.8836446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:43.8837264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:43.8838007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:43.8838778Z STAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8839712Z STAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8840015Z 2022-08-17T13:31:43.8846327Z STAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8846669Z 2022-08-17T13:31:43.8846953Z STAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8855650Z STAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8856031Z 2022-08-17T13:31:43.8860956Z STAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8865907Z STAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8866979Z STAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8867464Z 2022-08-17T13:31:43.8873377Z STAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8873703Z 2022-08-17T13:31:43.8879276Z STAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:43.8883402Z STAGE:2022-08-17 13:31:43 16226:16226 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8883888Z STAGE:2022-08-17 13:31:43 16227:16227 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:43.8884427Z STAGE:2022-08-17 13:31:43 16228:16228 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:44.2820693Z ok (2.175s) 2022-08-17T13:31:44.2820919Z 2022-08-17T13:31:44.2821486Z ---------------------------------------------------------------------- 2022-08-17T13:31:44.2821840Z Ran 1 test in 2.175s 2022-08-17T13:31:44.2821949Z 2022-08-17T13:31:44.2822019Z OK 2022-08-17T13:31:44.2822117Z 2022-08-17T13:31:44.2822287Z Generating XML reports... 2022-08-17T13:31:44.2856766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133142.xml 2022-08-17T13:31:45.5401909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:45.5402289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:45.5404391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:45.5404787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:45.8784204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:45.8794000Z 2022-08-17T13:31:45.8794181Z Running tests... 2022-08-17T13:31:45.8794816Z ---------------------------------------------------------------------- 2022-08-17T13:31:46.1444323Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16291 2022-08-17T13:31:46.1473017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16292 2022-08-17T13:31:46.1501967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16293 2022-08-17T13:31:47.1324404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:47.1324912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:47.1327109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:47.1327675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:47.1710940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:47.1711384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:47.1714250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:47.1714739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:47.1772580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:47.1773095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:47.1776902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:47.1777393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:47.5406689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:47.5507823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:47.5508631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:47.5509328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:47.5509900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:47.5510778Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:47.5517242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:47.5518380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:47.5519849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:47.5726496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:47.5828141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:47.5828857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:47.5830158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:47.5831123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:47.5831951Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:47.5832751Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5833496Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5834106Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5838641Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5839153Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5839629Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5848437Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5848917Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5854652Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5858947Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5859431Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5859876Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5865697Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5866296Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5872579Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:47.5877274Z STAGE:2022-08-17 13:31:47 16291:16291 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5878002Z STAGE:2022-08-17 13:31:47 16293:16293 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:47.5878487Z STAGE:2022-08-17 13:31:47 16292:16292 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:48.0555451Z ok (2.176s) 2022-08-17T13:31:48.0555691Z 2022-08-17T13:31:48.0556008Z ---------------------------------------------------------------------- 2022-08-17T13:31:48.0556274Z Ran 1 test in 2.176s 2022-08-17T13:31:48.0556400Z 2022-08-17T13:31:48.0556451Z OK 2022-08-17T13:31:48.0556547Z 2022-08-17T13:31:48.0556646Z Generating XML reports... 2022-08-17T13:31:48.0592215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133145.xml 2022-08-17T13:31:49.3625314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:49.3625709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:49.3628522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:49.3628942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:49.7072025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:49.7082081Z 2022-08-17T13:31:49.7082226Z Running tests... 2022-08-17T13:31:49.7082850Z ---------------------------------------------------------------------- 2022-08-17T13:31:49.9747946Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16356 2022-08-17T13:31:49.9776517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16357 2022-08-17T13:31:49.9804891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16358 2022-08-17T13:31:50.9394620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:50.9395264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:50.9396914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:50.9397515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:50.9471825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:50.9472462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:50.9475677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:50.9476359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:50.9608664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:50.9609267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:50.9611870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:50.9612488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:51.3300065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:51.3401711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:51.3402297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:51.3403380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:51.3404302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:51.3405098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:51.3509780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:51.4415013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:51.4415606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:51.4521637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:51.4623997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:51.4624822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:51.4625710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:51.4626277Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:51.4626959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:51.4627971Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4628939Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4629242Z 2022-08-17T13:31:51.4635202Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4635930Z STAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4636757Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4644516Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4645499Z STAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4652132Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4657161Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4657772Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4658248Z STAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4663959Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4664446Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4670903Z STAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:51.4675931Z STAGE:2022-08-17 13:31:51 16356:16356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4676435Z STAGE:2022-08-17 13:31:51 16357:16357 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.4676890Z STAGE:2022-08-17 13:31:51 16358:16358 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:51.8858693Z ok (2.177s) 2022-08-17T13:31:51.8859218Z 2022-08-17T13:31:51.8859555Z ---------------------------------------------------------------------- 2022-08-17T13:31:51.8859822Z Ran 1 test in 2.178s 2022-08-17T13:31:51.8859947Z 2022-08-17T13:31:51.8860013Z OK 2022-08-17T13:31:51.8860111Z 2022-08-17T13:31:51.8860192Z Generating XML reports... 2022-08-17T13:31:51.8895441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133149.xml 2022-08-17T13:31:53.1836275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:53.1836681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:53.1839536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:53.1839940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:53.5243876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:53.5253431Z 2022-08-17T13:31:53.5253566Z Running tests... 2022-08-17T13:31:53.5254228Z ---------------------------------------------------------------------- 2022-08-17T13:31:53.7870143Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16421 2022-08-17T13:31:53.7898559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16422 2022-08-17T13:31:53.7928004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16423 2022-08-17T13:31:54.7789775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:54.7790409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:54.7791727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:54.7792405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:54.7885270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:54.7886090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:54.7889488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:54.7890105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:54.7993801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:54.7994282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:54.7997427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:54.7997964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:55.1726860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:55.1762782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:55.1763272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:55.1763917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:55.1764478Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:55.1827656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:55.1872107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:55.1872652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:55.1978036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:55.1978451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:55.2839200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:55.2841746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:55.2842465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:55.2889772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:55.2890389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:55.2892815Z STAGE:2022-08-17 13:31:55 16422:16422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:55.2993986Z STAGE:2022-08-17 13:31:55 16423:16423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:55.3000255Z STAGE:2022-08-17 13:31:55 16422:16422 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:31:55 16423:16423 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:55.3000589Z 2022-08-17T13:31:55.3009532Z STAGE:2022-08-17 13:31:55 16423:16423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:55.3014891Z STAGE:2022-08-17 13:31:55 16422:16422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:55.3019569Z STAGE:2022-08-17 13:31:55 16422:16422 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:55.3020176Z STAGE:2022-08-17 13:31:55 16423:16423 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:55.6979057Z ok (2.172s) 2022-08-17T13:31:55.6979360Z 2022-08-17T13:31:55.6979844Z ---------------------------------------------------------------------- 2022-08-17T13:31:55.6980331Z Ran 1 test in 2.173s 2022-08-17T13:31:55.6980536Z 2022-08-17T13:31:55.6980639Z OK 2022-08-17T13:31:55.6980740Z 2022-08-17T13:31:55.6980836Z Generating XML reports... 2022-08-17T13:31:55.7015476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133153.xml 2022-08-17T13:31:56.9989028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:56.9989407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:56.9991564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:56.9991937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:57.3471927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:31:57.3481915Z 2022-08-17T13:31:57.3482341Z Running tests... 2022-08-17T13:31:57.3482868Z ---------------------------------------------------------------------- 2022-08-17T13:31:57.6136657Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16482 2022-08-17T13:31:57.6167613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16483 2022-08-17T13:31:57.6196383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16484 2022-08-17T13:31:58.5910829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:58.5911235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:58.5913932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:58.5914648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:58.6017922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:58.6018292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:58.6022499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:58.6022861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:58.6501143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:31:58.6501511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:31:58.6505387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:31:58.6505756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:31:59.0158836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:31:59.0260517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:31:59.0261007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:31:59.0261658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:59.0262290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:59.0262900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:31:59.0269361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:31:59.0269986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:31:59.0270581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:31:59.0271567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:31:59.0478864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:31:59.0479322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:31:59.0479973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:59.0480519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:59.0482613Z STAGE:2022-08-17 13:31:59 16483:16483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:59.0483077Z STAGE:2022-08-17 13:31:59 16484:16484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:59.0489486Z STAGE:2022-08-17 13:31:59 16484:16484 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:59.0489945Z STAGE:2022-08-17 13:31:59 16483:16483 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:59.0499602Z STAGE:2022-08-17 13:31:59 16484:16484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:59.0504784Z STAGE:2022-08-17 13:31:59 16483:16483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:31:59.0509297Z STAGE:2022-08-17 13:31:59 16483:16483 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:59.0509763Z STAGE:2022-08-17 13:31:59 16484:16484 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:31:59.0577253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:31:59.5248926Z ok (2.176s) 2022-08-17T13:31:59.5249191Z 2022-08-17T13:31:59.5249732Z ---------------------------------------------------------------------- 2022-08-17T13:31:59.5250110Z Ran 1 test in 2.177s 2022-08-17T13:31:59.5250233Z 2022-08-17T13:31:59.5250298Z OK 2022-08-17T13:31:59.5250393Z 2022-08-17T13:31:59.5250488Z Generating XML reports... 2022-08-17T13:31:59.5285426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133157.xml 2022-08-17T13:32:00.8463275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:00.8463692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:00.8466029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:00.8466425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:01.1964257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:01.1974562Z 2022-08-17T13:32:01.1974651Z Running tests... 2022-08-17T13:32:01.1976112Z ---------------------------------------------------------------------- 2022-08-17T13:32:01.4696693Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16543 2022-08-17T13:32:01.4727342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16544 2022-08-17T13:32:01.4754888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16545 2022-08-17T13:32:02.4774916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:02.4775587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:02.4778853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:02.4779440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:02.4793811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:02.4794197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:02.4798362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:02.4798897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:02.5361585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:02.5361967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:02.5365169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:02.5365534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:02.9023889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:02.9125161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:02.9125812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:02.9126684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:02.9127250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:02.9127800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:02.9134608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:02.9135858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:02.9136870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:02.9138973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:32:02.9242600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:32:02.9243182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:32:02.9244084Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:02.9245049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:02.9246922Z STAGE:2022-08-17 13:32:02 16544:16544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:02.9343670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:02.9346857Z STAGE:2022-08-17 13:32:02 16545:16545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:02.9353849Z STAGE:2022-08-17 13:32:02 16545:16545 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:02.9354390Z STAGE:2022-08-17 13:32:02 16544:16544 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:02.9365774Z STAGE:2022-08-17 13:32:02 16545:16545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:02.9371953Z STAGE:2022-08-17 13:32:02 16544:16544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:02.9377255Z STAGE:2022-08-17 13:32:02 16544:16544 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:02 16545:16545 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:02.9377585Z 2022-08-17T13:32:03.3809888Z ok (2.183s) 2022-08-17T13:32:03.3810056Z 2022-08-17T13:32:03.3810438Z ---------------------------------------------------------------------- 2022-08-17T13:32:03.3810710Z Ran 1 test in 2.183s 2022-08-17T13:32:03.3810838Z 2022-08-17T13:32:03.3810902Z OK 2022-08-17T13:32:03.3810998Z 2022-08-17T13:32:03.3811093Z Generating XML reports... 2022-08-17T13:32:03.3845458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133201.xml 2022-08-17T13:32:04.6701475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:04.6702254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:04.6704844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:04.6705435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:05.0084194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:05.0094458Z 2022-08-17T13:32:05.0094886Z Running tests... 2022-08-17T13:32:05.0095571Z ---------------------------------------------------------------------- 2022-08-17T13:32:05.2717476Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16604 2022-08-17T13:32:05.2746190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16605 2022-08-17T13:32:05.2773994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16606 2022-08-17T13:32:06.2808431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:06.2809085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:06.2811313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:06.2811711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:06.3209715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:06.3210326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:06.3213593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:06.3214006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:06.3480840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:06.3481305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:06.3483921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:06.3484440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:06.7196068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:06.7296918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:06.7297658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:06.7298316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:06.7298880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:06.7299442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:06.7404132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:06.8309532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:06.8310186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:06.8311036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:32:06.8416165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:32:06.8417030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:06.8417658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:32:06.8418496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:06.8421304Z STAGE:2022-08-17 13:32:06 16605:16605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:06.8515523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:06.8521045Z STAGE:2022-08-17 13:32:06 16606:16606 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:06.8527756Z STAGE:2022-08-17 13:32:06 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:06.8528371Z STAGE:2022-08-17 13:32:06 16605:16605 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:06.8536786Z STAGE:2022-08-17 13:32:06 16606:16606 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:06.8544090Z STAGE:2022-08-17 13:32:06 16605:16605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:06.8549475Z STAGE:2022-08-17 13:32:06 16605:16605 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:06 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:06.8549802Z 2022-08-17T13:32:07.2827191Z ok (2.273s) 2022-08-17T13:32:07.2827468Z 2022-08-17T13:32:07.2828008Z ---------------------------------------------------------------------- 2022-08-17T13:32:07.2828298Z Ran 1 test in 2.273s 2022-08-17T13:32:07.2828420Z 2022-08-17T13:32:07.2828485Z OK 2022-08-17T13:32:07.2828582Z 2022-08-17T13:32:07.2828677Z Generating XML reports... 2022-08-17T13:32:07.2863686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133205.xml 2022-08-17T13:32:08.5688338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:08.5688766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:08.5691008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:08.5691469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:08.9158023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:08.9168123Z 2022-08-17T13:32:08.9168375Z Running tests... 2022-08-17T13:32:08.9168799Z ---------------------------------------------------------------------- 2022-08-17T13:32:09.1848545Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16665 2022-08-17T13:32:09.1878818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16666 2022-08-17T13:32:09.1908398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16667 2022-08-17T13:32:10.1713796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:10.1714472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:10.1717193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:10.1717793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:10.1976346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:10.1977035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:10.1978676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:10.1979338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:10.2279337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:10.2279918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:10.2282861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:10.2283579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:10.5864710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:10.5964487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:10.5965037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:10.5965730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:10.5966472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:10.5967184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:10.6071903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:10.6075241Z STAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.6976715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:10.6977371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:10.6982077Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.6982753Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.6988806Z STAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.6989588Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.6990588Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.6998649Z STAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.6999253Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.7004720Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.7009388Z STAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.7010134Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.7010604Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.7017882Z STAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.7021354Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.7021850Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:10.7027487Z STAGE:2022-08-17 13:32:10 16665:16665 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.7028224Z STAGE:2022-08-17 13:32:10 16666:16666 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:10 16667:16667 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:10.7028529Z 2022-08-17T13:32:11.0960545Z ok (2.179s) 2022-08-17T13:32:11.0960723Z 2022-08-17T13:32:11.0961247Z ---------------------------------------------------------------------- 2022-08-17T13:32:11.0961707Z Ran 1 test in 2.179s 2022-08-17T13:32:11.0961893Z 2022-08-17T13:32:11.0961957Z OK 2022-08-17T13:32:11.0962052Z 2022-08-17T13:32:11.0962151Z Generating XML reports... 2022-08-17T13:32:11.0997418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133208.xml 2022-08-17T13:32:12.3809558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:12.3809937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:12.3812051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:12.3812487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:12.7197627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:12.7207522Z 2022-08-17T13:32:12.7207678Z Running tests... 2022-08-17T13:32:12.7208079Z ---------------------------------------------------------------------- 2022-08-17T13:32:12.9821138Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16721 2022-08-17T13:32:12.9848298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16722 2022-08-17T13:32:12.9875940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16723 2022-08-17T13:32:13.9555508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:13.9555909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:13.9557839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:13.9558331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:13.9558872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:13.9559227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:13.9563642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:13.9564017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:13.9687105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:13.9687768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:13.9690645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:13.9691311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:14.3313316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:14.3313899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:14.3314352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:14.3314988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:14.3315549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:14.3316093Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:14.3419188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:14.3422419Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4326144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:14.4326752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:14.4330739Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4331079Z 2022-08-17T13:32:14.4336358Z STAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4337143Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4345340Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4345996Z STAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4349025Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4354701Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4360039Z STAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4360928Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4361716Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4366965Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4367582Z STAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4372231Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:14.4377438Z STAGE:2022-08-17 13:32:14 16721:16721 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4378297Z STAGE:2022-08-17 13:32:14 16722:16722 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.4379358Z STAGE:2022-08-17 13:32:14 16723:16723 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:14.8927055Z ok (2.172s) 2022-08-17T13:32:14.8927308Z 2022-08-17T13:32:14.8927755Z ---------------------------------------------------------------------- 2022-08-17T13:32:14.8928037Z Ran 1 test in 2.172s 2022-08-17T13:32:14.8928160Z 2022-08-17T13:32:14.8928210Z OK 2022-08-17T13:32:14.8928311Z 2022-08-17T13:32:14.8928408Z Generating XML reports... 2022-08-17T13:32:14.8963254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133212.xml 2022-08-17T13:32:16.1797820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:16.1798423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:16.1803715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:16.1804491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:16.5184098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:16.5194812Z 2022-08-17T13:32:16.5195200Z Running tests... 2022-08-17T13:32:16.5195874Z ---------------------------------------------------------------------- 2022-08-17T13:32:16.5215992Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-08-17T13:32:16.5216424Z 2022-08-17T13:32:16.5216841Z ---------------------------------------------------------------------- 2022-08-17T13:32:16.5217291Z Ran 1 test in 0.002s 2022-08-17T13:32:16.5217504Z 2022-08-17T13:32:16.5217616Z OK (skipped=1) 2022-08-17T13:32:16.5217812Z 2022-08-17T13:32:16.5217966Z Generating XML reports... 2022-08-17T13:32:16.5242963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133216.xml 2022-08-17T13:32:17.6663782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:17.6664428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:17.6666893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:17.6667478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:18.0082158Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:18.0092477Z 2022-08-17T13:32:18.0092813Z Running tests... 2022-08-17T13:32:18.0093776Z ---------------------------------------------------------------------- 2022-08-17T13:32:18.2710991Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16787 2022-08-17T13:32:18.2738332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16788 2022-08-17T13:32:18.2766316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16789 2022-08-17T13:32:19.2612941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:19.2613375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:19.2615509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:19.2616056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:19.2656826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:19.2657505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:19.2660430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:19.2661382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:19.2686105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:19.2686751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:19.2690694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:19.2691369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:19.6342405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:19.6442831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:19.6443508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:19.6444337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:19.6445078Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:19.6445632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:19.6552028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:19.6552635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:19.6555606Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.6556101Z STAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.6561208Z STAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.6570923Z STAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7457090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:19.7459403Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7465796Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7466583Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7477039Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7481714Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7487217Z STAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7487885Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7488336Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7494095Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7494590Z STAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7499750Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:19.7504798Z STAGE:2022-08-17 13:32:19 16787:16787 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7505956Z STAGE:2022-08-17 13:32:19 16788:16788 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:19 16789:16789 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:19.7506507Z 2022-08-17T13:32:20.1817358Z ok (2.172s) 2022-08-17T13:32:20.1817650Z 2022-08-17T13:32:20.1818196Z ---------------------------------------------------------------------- 2022-08-17T13:32:20.1818514Z Ran 1 test in 2.172s 2022-08-17T13:32:20.1818639Z 2022-08-17T13:32:20.1818701Z OK 2022-08-17T13:32:20.1818804Z 2022-08-17T13:32:20.1818900Z Generating XML reports... 2022-08-17T13:32:20.1857075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133218.xml 2022-08-17T13:32:21.4715019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:21.4715419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:21.4717304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:21.4717675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:21.8148782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:21.8158773Z 2022-08-17T13:32:21.8158870Z Running tests... 2022-08-17T13:32:21.8159462Z ---------------------------------------------------------------------- 2022-08-17T13:32:22.0883293Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16843 2022-08-17T13:32:22.0912248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16844 2022-08-17T13:32:22.0940575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16845 2022-08-17T13:32:23.0778389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:23.0778988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:23.0781815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:23.0782511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:23.0939827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:23.0940454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:23.0943927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:23.0944591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:23.1154443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:23.1155494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:23.1159214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:23.1159711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:23.4891556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:23.4892098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:23.4892517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:23.4893362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:23.4894289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:23.4895088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:23.4901521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:23.4902016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:23.4902484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:23.4907476Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4907983Z STAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4908440Z STAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4914116Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4914618Z 2022-08-17T13:32:23.4915097Z STAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4926269Z STAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4926763Z 2022-08-17T13:32:23.4928955Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4934347Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4934747Z 2022-08-17T13:32:23.4935032Z STAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4940025Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4942316Z STAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4948091Z STAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:23.4953291Z STAGE:2022-08-17 13:32:23 16843:16843 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4953773Z STAGE:2022-08-17 13:32:23 16844:16844 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.4954221Z STAGE:2022-08-17 13:32:23 16845:16845 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:23.8991280Z ok (2.083s) 2022-08-17T13:32:23.8991470Z 2022-08-17T13:32:23.8991791Z ---------------------------------------------------------------------- 2022-08-17T13:32:23.8992291Z Ran 1 test in 2.083s 2022-08-17T13:32:23.8992474Z 2022-08-17T13:32:23.8992574Z OK 2022-08-17T13:32:23.8992726Z 2022-08-17T13:32:23.8992886Z Generating XML reports... 2022-08-17T13:32:23.9027955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133221.xml 2022-08-17T13:32:25.1889229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:25.1889613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:25.1892002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:25.1892381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:25.5307232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:25.5317211Z 2022-08-17T13:32:25.5317695Z Running tests... 2022-08-17T13:32:25.5318108Z ---------------------------------------------------------------------- 2022-08-17T13:32:25.5338373Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-08-17T13:32:25.5338798Z 2022-08-17T13:32:25.5339516Z ---------------------------------------------------------------------- 2022-08-17T13:32:25.5339792Z Ran 1 test in 0.002s 2022-08-17T13:32:25.5339911Z 2022-08-17T13:32:25.5339989Z OK (skipped=1) 2022-08-17T13:32:25.5340105Z 2022-08-17T13:32:25.5340195Z Generating XML reports... 2022-08-17T13:32:25.5365318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133225.xml 2022-08-17T13:32:26.6921163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:26.6921536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:26.6923968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:26.6924477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:27.0362225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:27.0373086Z 2022-08-17T13:32:27.0373315Z Running tests... 2022-08-17T13:32:27.0373955Z ---------------------------------------------------------------------- 2022-08-17T13:32:27.0395658Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-08-17T13:32:27.0396099Z 2022-08-17T13:32:27.0396332Z ---------------------------------------------------------------------- 2022-08-17T13:32:27.0396587Z Ran 1 test in 0.002s 2022-08-17T13:32:27.0396707Z 2022-08-17T13:32:27.0396785Z OK (skipped=1) 2022-08-17T13:32:27.0396885Z 2022-08-17T13:32:27.0396974Z Generating XML reports... 2022-08-17T13:32:27.0421556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133227.xml 2022-08-17T13:32:28.1906205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:28.1906611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:28.1909120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:28.1909502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:28.5283548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:28.5293272Z 2022-08-17T13:32:28.5293374Z Running tests... 2022-08-17T13:32:28.5293869Z ---------------------------------------------------------------------- 2022-08-17T13:32:28.7893362Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16919 2022-08-17T13:32:28.7921833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16920 2022-08-17T13:32:28.7951564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16921 2022-08-17T13:32:29.7726926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:29.7727640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:29.7728403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:29.7728953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:29.7731018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:29.7731584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:29.7732413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:29.7733029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:29.8089172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:29.8089827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:29.8092656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:29.8093354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:30.1789004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:30.1789445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:30.1789855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:30.1790506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:30.1791063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:30.1791616Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:30.1894910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:30.1898500Z STAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2799572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:30.2799993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:30.2804501Z STAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2805032Z STAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2812914Z STAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2813671Z STAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2814048Z 2022-08-17T13:32:30.2824434Z STAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2824974Z STAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2832092Z STAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2840450Z STAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2840768Z 2022-08-17T13:32:30.2841052Z STAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2852313Z STAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2852974Z STAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2856583Z STAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:30.2864491Z STAGE:2022-08-17 13:32:30 16919:16919 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2865607Z STAGE:2022-08-17 13:32:30 16921:16921 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:30 16920:16920 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:30.2866103Z 2022-08-17T13:32:30.7000854Z ok (2.170s) 2022-08-17T13:32:30.7001080Z 2022-08-17T13:32:30.7001842Z ---------------------------------------------------------------------- 2022-08-17T13:32:30.7002258Z Ran 1 test in 2.171s 2022-08-17T13:32:30.7002455Z 2022-08-17T13:32:30.7002555Z OK 2022-08-17T13:32:30.7002710Z 2022-08-17T13:32:30.7002839Z Generating XML reports... 2022-08-17T13:32:30.7038306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133228.xml 2022-08-17T13:32:31.9949280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:31.9949685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:31.9952307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:31.9952790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:32.3443372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:32.3453420Z 2022-08-17T13:32:32.3453539Z Running tests... 2022-08-17T13:32:32.3454146Z ---------------------------------------------------------------------- 2022-08-17T13:32:32.6156973Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16975 2022-08-17T13:32:32.6186492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16976 2022-08-17T13:32:32.6217485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16977 2022-08-17T13:32:33.6051611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:33.6052290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:33.6055219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:33.6055629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:33.6479522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:33.6479893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:33.6483329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:33.6483709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:33.6720273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:33.6720619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:33.6724252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:33.6724786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:34.0324994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:34.0426170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:34.0426953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:34.0427619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:34.0428181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:34.0428735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:34.0533921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:34.0537866Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1438145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:34.1438560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:34.1441545Z STAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1442468Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1448883Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1449678Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1450436Z STAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1461267Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1463753Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1464064Z 2022-08-17T13:32:34.1467834Z STAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1468646Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1469100Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1474686Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1475098Z 2022-08-17T13:32:34.1475585Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:34.1477987Z STAGE:2022-08-17 13:32:34 16977:16977 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1478760Z STAGE:2022-08-17 13:32:34 16975:16975 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.1479672Z STAGE:2022-08-17 13:32:34 16976:16976 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:34.5272337Z ok (2.181s) 2022-08-17T13:32:34.5272579Z 2022-08-17T13:32:34.5273110Z ---------------------------------------------------------------------- 2022-08-17T13:32:34.5273447Z Ran 1 test in 2.182s 2022-08-17T13:32:34.5273570Z 2022-08-17T13:32:34.5273635Z OK 2022-08-17T13:32:34.5273932Z 2022-08-17T13:32:34.5274030Z Generating XML reports... 2022-08-17T13:32:34.5308686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133232.xml 2022-08-17T13:32:35.8268483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:35.8268876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:35.8271483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:35.8271896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:36.1667361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:36.1676922Z 2022-08-17T13:32:36.1677156Z Running tests... 2022-08-17T13:32:36.1677580Z ---------------------------------------------------------------------- 2022-08-17T13:32:36.4376627Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17031 2022-08-17T13:32:36.4404113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17032 2022-08-17T13:32:36.4432649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17033 2022-08-17T13:32:37.4461324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:37.4461962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:37.4464281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:37.4464897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:37.4512416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:37.4513099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:37.4517147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:37.4517778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:37.4674826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:37.4675510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:37.4679212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:37.4679793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:37.8492781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:37.8593864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:37.8594538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:37.8595547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:37.8596244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:37.8596885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:37.8602733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:37.8603991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:37.8604518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:38.1478505Z ok (1.980s) 2022-08-17T13:32:38.1478782Z 2022-08-17T13:32:38.1479286Z ---------------------------------------------------------------------- 2022-08-17T13:32:38.1479544Z Ran 1 test in 1.980s 2022-08-17T13:32:38.1479665Z 2022-08-17T13:32:38.1479731Z OK 2022-08-17T13:32:38.1479828Z 2022-08-17T13:32:38.1479925Z Generating XML reports... 2022-08-17T13:32:38.1514142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133236.xml 2022-08-17T13:32:39.4385601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:39.4386006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:39.4388907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:39.7884361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:39.7884900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:39.7894354Z 2022-08-17T13:32:39.7894478Z Running tests... 2022-08-17T13:32:39.7895088Z ---------------------------------------------------------------------- 2022-08-17T13:32:40.0543841Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17084 2022-08-17T13:32:40.0572142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17085 2022-08-17T13:32:40.0599732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17086 2022-08-17T13:32:41.0349134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:41.0349796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:41.0351839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:41.0352475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:41.0377677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:41.0378291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:41.0382649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:41.0383279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:41.0694655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:41.0695346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:41.0698965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:41.0699619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:41.4319204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:41.4319832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:41.4320243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:41.4320895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:41.4321465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:41.4322005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:41.4327571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:41.4328372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:41.4329232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:41.4332316Z STAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4333328Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4333763Z 2022-08-17T13:32:41.4339624Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4340414Z STAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4341170Z STAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4352766Z STAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4353447Z STAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4354190Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4357190Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4358348Z STAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4358824Z 2022-08-17T13:32:41.4363687Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4364448Z STAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4365365Z STAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:41.4369006Z STAGE:2022-08-17 13:32:41 17084:17084 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4369650Z STAGE:2022-08-17 13:32:41 17086:17086 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:41 17085:17085 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:41.4369958Z 2022-08-17T13:32:41.8649485Z ok (2.075s) 2022-08-17T13:32:41.8649709Z 2022-08-17T13:32:41.8650174Z ---------------------------------------------------------------------- 2022-08-17T13:32:41.8650603Z Ran 1 test in 2.075s 2022-08-17T13:32:41.8650796Z 2022-08-17T13:32:41.8650896Z OK 2022-08-17T13:32:41.8651040Z 2022-08-17T13:32:41.8651187Z Generating XML reports... 2022-08-17T13:32:41.8686782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133239.xml 2022-08-17T13:32:43.1695046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:43.1695475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:43.1697741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:43.1698124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:43.5141948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:43.5152103Z 2022-08-17T13:32:43.5152306Z Running tests... 2022-08-17T13:32:43.5152715Z ---------------------------------------------------------------------- 2022-08-17T13:32:43.5171114Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:32:43.5171519Z 2022-08-17T13:32:43.5171991Z ---------------------------------------------------------------------- 2022-08-17T13:32:43.5172569Z Ran 1 test in 0.002s 2022-08-17T13:32:43.5172677Z 2022-08-17T13:32:43.5172756Z OK (skipped=1) 2022-08-17T13:32:43.5172870Z 2022-08-17T13:32:43.5172963Z Generating XML reports... 2022-08-17T13:32:43.5197967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133243.xml 2022-08-17T13:32:44.6757549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:44.6757940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:44.6760026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:44.6760403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:45.0199786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:45.0210279Z 2022-08-17T13:32:45.0210677Z Running tests... 2022-08-17T13:32:45.0211092Z ---------------------------------------------------------------------- 2022-08-17T13:32:45.0231142Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:32:45.0231611Z 2022-08-17T13:32:45.0232089Z ---------------------------------------------------------------------- 2022-08-17T13:32:45.0232361Z Ran 1 test in 0.002s 2022-08-17T13:32:45.0232481Z 2022-08-17T13:32:45.0232555Z OK (skipped=1) 2022-08-17T13:32:45.0232655Z 2022-08-17T13:32:45.0232746Z Generating XML reports... 2022-08-17T13:32:45.0256841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133245.xml 2022-08-17T13:32:46.1760567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:46.1760944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:46.1763718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:46.1764102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:46.5165000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:46.5175775Z 2022-08-17T13:32:46.5176057Z Running tests... 2022-08-17T13:32:46.5176705Z ---------------------------------------------------------------------- 2022-08-17T13:32:46.7836347Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17160 2022-08-17T13:32:46.7865104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17161 2022-08-17T13:32:46.7893444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17162 2022-08-17T13:32:47.7654912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:47.7655611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:47.7659345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:47.7659819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:47.7729269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:47.7729717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:47.7733687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:47.7734197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:47.7903215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:47.7903900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:47.7907656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:47.7908334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:48.1466245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:48.1566762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:48.1567482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:48.1568379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:48.1568939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:48.1569503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:48.1574324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:48.1576077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:48.1576995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:48.1884448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:32:48.1885196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:32:48.1885596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:32:48.1886696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:48.1887482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:48.1888150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:48.1888929Z STAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.1889661Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.1894528Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.1905440Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.1988879Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.1993607Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.1994105Z STAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.2005439Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.2006082Z 2022-08-17T13:32:48.2010309Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.2010823Z 2022-08-17T13:32:48.2011242Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.2015848Z STAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.2016714Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.2017178Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:48.2021343Z STAGE:2022-08-17 13:32:48 17160:17160 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.2022047Z STAGE:2022-08-17 13:32:48 17161:17161 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.2022626Z STAGE:2022-08-17 13:32:48 17162:17162 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:48.5940686Z ok (2.076s) 2022-08-17T13:32:48.5940898Z 2022-08-17T13:32:48.5941330Z ---------------------------------------------------------------------- 2022-08-17T13:32:48.5941686Z Ran 1 test in 2.076s 2022-08-17T13:32:48.5941810Z 2022-08-17T13:32:48.5941877Z OK 2022-08-17T13:32:48.5942023Z 2022-08-17T13:32:48.5942276Z Generating XML reports... 2022-08-17T13:32:48.5980176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133246.xml 2022-08-17T13:32:49.8857346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:49.8858274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:49.8859925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:49.8860569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:50.2282804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:50.2292913Z 2022-08-17T13:32:50.2293245Z Running tests... 2022-08-17T13:32:50.2293896Z ---------------------------------------------------------------------- 2022-08-17T13:32:50.4942614Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17225 2022-08-17T13:32:50.4970781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17226 2022-08-17T13:32:50.4999103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17227 2022-08-17T13:32:51.4894622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:51.4895010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:51.4898359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:51.4898865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:51.5390432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:51.5390791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:51.5394183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:51.5394560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:51.5426041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:51.5426389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:51.5430704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:51.5431080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:51.9011479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:51.9113076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:51.9113971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:51.9114965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:51.9115707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:51.9116284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:51.9122277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:51.9122808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:51.9124586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:32:51.9124983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:51.9330401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:32:51.9330821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:32:51.9331615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:51.9332166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:51.9334256Z STAGE:2022-08-17 13:32:51 17226:17226 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:32:51 17227:17227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:51.9334564Z 2022-08-17T13:32:51.9339377Z STAGE:2022-08-17 13:32:51 17226:17226 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:51.9339859Z STAGE:2022-08-17 13:32:51 17227:17227 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:51.9350933Z STAGE:2022-08-17 13:32:51 17227:17227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:51.9351516Z STAGE:2022-08-17 13:32:51 17226:17226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:32:51.9354725Z STAGE:2022-08-17 13:32:51 17226:17226 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:51.9355197Z STAGE:2022-08-17 13:32:51 17227:17227 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:32:51.9429480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:32:52.4046996Z ok (2.175s) 2022-08-17T13:32:52.4047243Z 2022-08-17T13:32:52.4047699Z ---------------------------------------------------------------------- 2022-08-17T13:32:52.4048099Z Ran 1 test in 2.175s 2022-08-17T13:32:52.4048298Z 2022-08-17T13:32:52.4048374Z OK 2022-08-17T13:32:52.4048532Z 2022-08-17T13:32:52.4051410Z Generating XML reports... 2022-08-17T13:32:52.4083876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133250.xml 2022-08-17T13:32:53.7120645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:53.7121137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:53.7123845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:53.7124232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:54.0580765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:54.0590597Z 2022-08-17T13:32:54.0590703Z Running tests... 2022-08-17T13:32:54.0591284Z ---------------------------------------------------------------------- 2022-08-17T13:32:54.3185652Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17286 2022-08-17T13:32:54.3214395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17287 2022-08-17T13:32:54.3243180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17288 2022-08-17T13:32:55.3053038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:55.3053730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:55.3057288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:55.3057941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:55.3203867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:55.3204516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:55.3207465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:55.3208388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:55.3258667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:55.3259326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:55.3262823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:55.3263224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:55.6957324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:55.7058055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:55.7058781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:55.7059646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:55.7060208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:55.7060751Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:55.7067382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:55.7067942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:55.7068913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:56.0290856Z ok (1.969s) 2022-08-17T13:32:56.0291167Z 2022-08-17T13:32:56.0291585Z ---------------------------------------------------------------------- 2022-08-17T13:32:56.0291850Z Ran 1 test in 1.970s 2022-08-17T13:32:56.0291984Z 2022-08-17T13:32:56.0292087Z OK 2022-08-17T13:32:56.0292195Z 2022-08-17T13:32:56.0292294Z Generating XML reports... 2022-08-17T13:32:56.0325429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133254.xml 2022-08-17T13:32:57.2877432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:57.2877837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:57.2880797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:57.2881176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:57.6315891Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:32:57.6326465Z 2022-08-17T13:32:57.6326588Z Running tests... 2022-08-17T13:32:57.6327029Z ---------------------------------------------------------------------- 2022-08-17T13:32:57.8948255Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17339 2022-08-17T13:32:57.8975345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17340 2022-08-17T13:32:57.9003759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17341 2022-08-17T13:32:58.8896035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:58.8896628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:58.8898295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:58.8898950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:58.8968992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:58.8969629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:58.8972861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:58.8973402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:58.9040818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:32:58.9041433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:32:58.9044868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:32:58.9045531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:32:59.2699951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:32:59.2799303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:32:59.2799933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:32:59.2800876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:59.2801799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:59.2802754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:32:59.2907916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:32:59.2908461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:32:59.3812399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:32:59.7051849Z ok (2.072s) 2022-08-17T13:32:59.7052080Z 2022-08-17T13:32:59.7052635Z ---------------------------------------------------------------------- 2022-08-17T13:32:59.7053019Z Ran 1 test in 2.072s 2022-08-17T13:32:59.7053202Z 2022-08-17T13:32:59.7053266Z OK 2022-08-17T13:32:59.7053364Z 2022-08-17T13:32:59.7053459Z Generating XML reports... 2022-08-17T13:32:59.7087818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133257.xml 2022-08-17T13:33:01.0183208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:01.0183589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:01.0186128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:01.0186508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:01.3663239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:01.3672818Z 2022-08-17T13:33:01.3672939Z Running tests... 2022-08-17T13:33:01.3674003Z ---------------------------------------------------------------------- 2022-08-17T13:33:01.6354485Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17392 2022-08-17T13:33:01.6384012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17393 2022-08-17T13:33:01.6413454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17394 2022-08-17T13:33:02.6282938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:02.6283623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:02.6285301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:02.6286457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:02.6449457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:02.6449854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:02.6452464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:02.6452936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:02.6520341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:02.6520790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:02.6524695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:02.6525211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:03.0232458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:03.0333788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:03.0334283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:03.0335134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:03.0336062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:03.0336698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:03.0342417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:03.0342984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:03.0343952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:03.3458816Z ok (1.978s) 2022-08-17T13:33:03.3459069Z 2022-08-17T13:33:03.3459512Z ---------------------------------------------------------------------- 2022-08-17T13:33:03.3459919Z Ran 1 test in 1.979s 2022-08-17T13:33:03.3460106Z 2022-08-17T13:33:03.3460201Z OK 2022-08-17T13:33:03.3460347Z 2022-08-17T13:33:03.3460474Z Generating XML reports... 2022-08-17T13:33:03.3496633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133301.xml 2022-08-17T13:33:04.6374352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:04.6374961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:04.6377366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:04.6377958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:04.9808727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:04.9819365Z 2022-08-17T13:33:04.9819926Z Running tests... 2022-08-17T13:33:04.9820377Z ---------------------------------------------------------------------- 2022-08-17T13:33:05.2476052Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17445 2022-08-17T13:33:05.2504393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17446 2022-08-17T13:33:05.2532532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17447 2022-08-17T13:33:06.2321129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:06.2321751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:06.2322447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:06.2323016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:06.2324121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:06.2324736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:06.2326168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:06.2326846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:06.2356306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:06.2356953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:06.2360818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:06.2361463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:06.6149902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:06.6251499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:06.6252265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:06.6252913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:06.6253486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:06.6254044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:06.6259681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:06.6260385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:06.6261354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:06.6262739Z STAGE:2022-08-17 13:33:06 17445:17445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:06.6263867Z STAGE:2022-08-17 13:33:06 17447:17447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:06.6265123Z STAGE:2022-08-17 13:33:06 17446:17446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:06.6284219Z STAGE:2022-08-17 13:33:06 17445:17445 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:06.6286682Z STAGE:2022-08-17 13:33:06 17446:17446 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:06.6287354Z STAGE:2022-08-17 13:33:06 17447:17447 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:07.0580508Z ok (2.076s) 2022-08-17T13:33:07.0580855Z 2022-08-17T13:33:07.0581176Z ---------------------------------------------------------------------- 2022-08-17T13:33:07.0581482Z Ran 1 test in 2.076s 2022-08-17T13:33:07.0581615Z 2022-08-17T13:33:07.0581680Z OK 2022-08-17T13:33:07.0581777Z 2022-08-17T13:33:07.0581875Z Generating XML reports... 2022-08-17T13:33:07.0616156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133304.xml 2022-08-17T13:33:08.3441959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:08.3442359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:08.3444520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:08.3444889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:08.6888237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:08.6897656Z 2022-08-17T13:33:08.6897799Z Running tests... 2022-08-17T13:33:08.6898376Z ---------------------------------------------------------------------- 2022-08-17T13:33:08.9562515Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17501 2022-08-17T13:33:08.9591447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17502 2022-08-17T13:33:08.9620497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17503 2022-08-17T13:33:09.9490228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:09.9490906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:09.9493165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:09.9493786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:09.9585336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:09.9586035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:09.9589674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:09.9590289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:09.9618339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:09.9618945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:09.9622549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:09.9623283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:10.3425625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:10.3527157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:10.3527936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:10.3528789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:10.3529344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:10.3529895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:10.3536799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:10.3541564Z STAGE:2022-08-17 13:33:10 17502:17502 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:10.4542234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:10.4547332Z STAGE:2022-08-17 13:33:10 17503:17503 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:10.4560079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:10.4566371Z STAGE:2022-08-17 13:33:10 17501:17501 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:10.4607500Z STAGE:2022-08-17 13:33:10 17502:17502 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:10.4608165Z STAGE:2022-08-17 13:33:10 17503:17503 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:33:10 17501:17501 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:10.4608481Z 2022-08-17T13:33:10.8670266Z ok (2.177s) 2022-08-17T13:33:10.8670526Z 2022-08-17T13:33:10.8671103Z ---------------------------------------------------------------------- 2022-08-17T13:33:10.8671440Z Ran 1 test in 2.177s 2022-08-17T13:33:10.8671563Z 2022-08-17T13:33:10.8671626Z OK 2022-08-17T13:33:10.8671721Z 2022-08-17T13:33:10.8671815Z Generating XML reports... 2022-08-17T13:33:10.8706753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133308.xml 2022-08-17T13:33:12.1718800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:12.1719214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:12.1722048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:12.1722408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:12.5179748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:12.5190320Z 2022-08-17T13:33:12.5190754Z Running tests... 2022-08-17T13:33:12.5191178Z ---------------------------------------------------------------------- 2022-08-17T13:33:12.7852266Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17557 2022-08-17T13:33:12.7879505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17558 2022-08-17T13:33:12.7908036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17559 2022-08-17T13:33:13.8384169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:13.8384826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:13.8387008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:13.8387676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:13.8688521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:13.8689173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:13.8691893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:13.8692536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:13.8954188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:13.8954863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:13.8958672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:13.8959306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:14.2687837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:14.2789061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:14.2789575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:14.2790281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:14.2790996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:14.2791552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:14.2895788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:14.2899092Z STAGE:2022-08-17 13:33:14 17558:17558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:14.3799988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:14.3800564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:14.3803935Z STAGE:2022-08-17 13:33:14 17557:17557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:14.3804950Z STAGE:2022-08-17 13:33:14 17559:17559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:14.3830966Z STAGE:2022-08-17 13:33:14 17559:17559 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:14.3832869Z STAGE:2022-08-17 13:33:14 17558:17558 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:14.3833766Z STAGE:2022-08-17 13:33:14 17557:17557 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:14.7962999Z ok (2.277s) 2022-08-17T13:33:14.7963173Z 2022-08-17T13:33:14.7963495Z ---------------------------------------------------------------------- 2022-08-17T13:33:14.7963762Z Ran 1 test in 2.277s 2022-08-17T13:33:14.7963885Z 2022-08-17T13:33:14.7963955Z OK 2022-08-17T13:33:14.7964052Z 2022-08-17T13:33:14.7964147Z Generating XML reports... 2022-08-17T13:33:14.7998209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133312.xml 2022-08-17T13:33:16.0742633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:16.0743137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:16.0745985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:16.0746529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:16.4185883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:16.4197041Z 2022-08-17T13:33:16.4197152Z Running tests... 2022-08-17T13:33:16.4198204Z ---------------------------------------------------------------------- 2022-08-17T13:33:16.4214596Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:33:16.4215160Z 2022-08-17T13:33:16.4215411Z ---------------------------------------------------------------------- 2022-08-17T13:33:16.4215655Z Ran 1 test in 0.002s 2022-08-17T13:33:16.4215774Z 2022-08-17T13:33:16.4215850Z OK (skipped=1) 2022-08-17T13:33:16.4215966Z 2022-08-17T13:33:16.4216062Z Generating XML reports... 2022-08-17T13:33:16.4240411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133316.xml 2022-08-17T13:33:17.5763037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:17.5763442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:17.5773627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:17.5774236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:17.9157416Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:17.9167421Z 2022-08-17T13:33:17.9167562Z Running tests... 2022-08-17T13:33:17.9168186Z ---------------------------------------------------------------------- 2022-08-17T13:33:17.9186496Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:33:17.9186963Z 2022-08-17T13:33:17.9187398Z ---------------------------------------------------------------------- 2022-08-17T13:33:17.9187815Z Ran 1 test in 0.002s 2022-08-17T13:33:17.9188020Z 2022-08-17T13:33:17.9188149Z OK (skipped=1) 2022-08-17T13:33:17.9188347Z 2022-08-17T13:33:17.9188488Z Generating XML reports... 2022-08-17T13:33:17.9213460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133317.xml 2022-08-17T13:33:19.0699837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:19.0700251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:19.0703151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:19.0703543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:19.4078222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:19.4087645Z 2022-08-17T13:33:19.4087744Z Running tests... 2022-08-17T13:33:19.4088718Z ---------------------------------------------------------------------- 2022-08-17T13:33:19.4108057Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:33:19.4108480Z 2022-08-17T13:33:19.4108957Z ---------------------------------------------------------------------- 2022-08-17T13:33:19.4109412Z Ran 1 test in 0.002s 2022-08-17T13:33:19.4109653Z 2022-08-17T13:33:19.4109767Z OK (skipped=1) 2022-08-17T13:33:19.4109966Z 2022-08-17T13:33:19.4110123Z Generating XML reports... 2022-08-17T13:33:19.4134849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133319.xml 2022-08-17T13:33:20.5582946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:20.5583322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:20.5585927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:20.5586416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:20.9016307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:20.9026137Z 2022-08-17T13:33:20.9026240Z Running tests... 2022-08-17T13:33:20.9026958Z ---------------------------------------------------------------------- 2022-08-17T13:33:21.1649288Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17643 2022-08-17T13:33:21.1676374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17644 2022-08-17T13:33:21.1703800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17645 2022-08-17T13:33:22.1712069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:22.1712793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:22.1715735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:22.1716391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:22.1816486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:22.1817151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:22.1820048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:22.1820963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:22.1875495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:22.1876163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:22.1880580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:22.1881186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:22.5490747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:22.5546256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:22.5546697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:22.5547368Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:22.5547917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:22.5592277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:22.5654322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:22.5661262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:22.5662042Z STAGE:2022-08-17 13:33:22 17645:17645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:22.5662519Z STAGE:2022-08-17 13:33:22 17644:17644 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:22.6603934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:22.6608356Z STAGE:2022-08-17 13:33:22 17643:17643 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:22.6635934Z STAGE:2022-08-17 13:33:22 17645:17645 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:22.6636786Z STAGE:2022-08-17 13:33:22 17644:17644 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:22.6637604Z STAGE:2022-08-17 13:33:22 17643:17643 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:23.0752788Z ok (2.172s) 2022-08-17T13:33:23.0753036Z 2022-08-17T13:33:23.0753564Z ---------------------------------------------------------------------- 2022-08-17T13:33:23.0754090Z Ran 1 test in 2.173s 2022-08-17T13:33:23.0754215Z 2022-08-17T13:33:23.0754266Z OK 2022-08-17T13:33:23.0754362Z 2022-08-17T13:33:23.0754458Z Generating XML reports... 2022-08-17T13:33:23.0789505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133320.xml 2022-08-17T13:33:24.3626520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:24.3627214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:24.3629742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:24.3630332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:24.7073686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:24.7084572Z 2022-08-17T13:33:24.7084826Z Running tests... 2022-08-17T13:33:24.7085462Z ---------------------------------------------------------------------- 2022-08-17T13:33:24.9757714Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17699 2022-08-17T13:33:24.9786102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17700 2022-08-17T13:33:24.9814166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17701 2022-08-17T13:33:25.9599251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:25.9599886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:25.9602967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:25.9603597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:25.9885750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:25.9886615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:25.9889149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:25.9889815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:26.0019892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:26.0020623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:26.0023247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:26.0023939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:26.3702687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:26.3727871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:26.3728352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:26.3729031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:26.3729577Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:26.3803523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:26.3836660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:26.3837360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:26.4814961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:26.7861309Z ok (2.077s) 2022-08-17T13:33:26.7861819Z 2022-08-17T13:33:26.7862600Z ---------------------------------------------------------------------- 2022-08-17T13:33:26.7862971Z Ran 1 test in 2.078s 2022-08-17T13:33:26.7863095Z 2022-08-17T13:33:26.7863183Z OK 2022-08-17T13:33:26.7863282Z 2022-08-17T13:33:26.7863380Z Generating XML reports... 2022-08-17T13:33:26.7897655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133324.xml 2022-08-17T13:33:28.1174849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:28.1175255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:28.1177964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:28.1178443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:28.4628176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:28.4637893Z 2022-08-17T13:33:28.4637995Z Running tests... 2022-08-17T13:33:28.4638837Z ---------------------------------------------------------------------- 2022-08-17T13:33:28.7318523Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17752 2022-08-17T13:33:28.7347454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17753 2022-08-17T13:33:28.7375365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17754 2022-08-17T13:33:29.7272978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:29.7273647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:29.7275962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:29.7276621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:29.7372514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:29.7373218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:29.7377148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:29.7377591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:29.7383530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:29.7384033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:29.7387947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:29.7388510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:30.0983585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:30.1074466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:30.1074908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:30.1075617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:30.1076163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:30.1084915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:30.1182389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:30.1182878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:30.1187200Z STAGE:2022-08-17 13:33:30 17754:17754 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:30.1187707Z STAGE:2022-08-17 13:33:30 17753:17753 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:30.2095821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:30.2099454Z STAGE:2022-08-17 13:33:30 17752:17752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:30.2114779Z STAGE:2022-08-17 13:33:30 17752:17752 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:30.2117479Z STAGE:2022-08-17 13:33:30 17753:17753 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:30.2118412Z STAGE:2022-08-17 13:33:30 17754:17754 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:30.6427599Z ok (2.179s) 2022-08-17T13:33:30.6427885Z 2022-08-17T13:33:30.6428379Z ---------------------------------------------------------------------- 2022-08-17T13:33:30.6428663Z Ran 1 test in 2.179s 2022-08-17T13:33:30.6429005Z 2022-08-17T13:33:30.6429070Z OK 2022-08-17T13:33:30.6429169Z 2022-08-17T13:33:30.6429268Z Generating XML reports... 2022-08-17T13:33:30.6463787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133328.xml 2022-08-17T13:33:31.9415271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:31.9415675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:31.9417928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:31.9418381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:32.2889280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:32.2899288Z 2022-08-17T13:33:32.2899419Z Running tests... 2022-08-17T13:33:32.2899964Z ---------------------------------------------------------------------- 2022-08-17T13:33:32.5579985Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17808 2022-08-17T13:33:32.5609157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17809 2022-08-17T13:33:32.5637391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17810 2022-08-17T13:33:33.5438457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:33.5439126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:33.5441563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:33.5442204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:33.5558147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:33.5558542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:33.5561905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:33.5562459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:33.5799109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:33.5799487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:33.5802786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:33.5803211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:33.9412721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:33.9469839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:33.9470512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:33.9471161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:33.9471716Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:33.9513307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:33.9577760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:33.9578472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:33.9583734Z STAGE:2022-08-17 13:33:33 17809:17809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:33.9584312Z STAGE:2022-08-17 13:33:33 17810:17810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:34.0524813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:34.0529809Z STAGE:2022-08-17 13:33:34 17808:17808 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:33:34.0545829Z STAGE:2022-08-17 13:33:34 17808:17808 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:34.0549185Z STAGE:2022-08-17 13:33:34 17809:17809 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:34.0549747Z STAGE:2022-08-17 13:33:34 17810:17810 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:33:34.4688978Z ok (2.179s) 2022-08-17T13:33:34.4689201Z 2022-08-17T13:33:34.4689652Z ---------------------------------------------------------------------- 2022-08-17T13:33:34.4690082Z Ran 1 test in 2.179s 2022-08-17T13:33:34.4690279Z 2022-08-17T13:33:34.4690388Z OK 2022-08-17T13:33:34.4690537Z 2022-08-17T13:33:34.4690686Z Generating XML reports... 2022-08-17T13:33:34.4726476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133332.xml 2022-08-17T13:33:35.8209528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:35.8209939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:35.8212333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:35.8212802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:36.1646084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:36.1656405Z 2022-08-17T13:33:36.1656786Z Running tests... 2022-08-17T13:33:36.1657184Z ---------------------------------------------------------------------- 2022-08-17T13:33:36.4373691Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17864 2022-08-17T13:33:36.4402460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17865 2022-08-17T13:33:36.4431670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17866 2022-08-17T13:33:37.4657464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:37.4658443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:37.4660986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:37.4661588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:37.4682125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:37.4682730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:37.4687104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:37.4687682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:37.4972454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:37.4973165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:37.4977251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:37.4977864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:37.8689303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:37.8689769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:37.8690163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:37.8690800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:37.8691360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:37.8691893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:37.8697360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:37.8698567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:37.9701378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:38.3480569Z ok (2.182s) 2022-08-17T13:33:38.3480793Z 2022-08-17T13:33:38.3481134Z ---------------------------------------------------------------------- 2022-08-17T13:33:38.3481404Z Ran 1 test in 2.182s 2022-08-17T13:33:38.3481525Z 2022-08-17T13:33:38.3481591Z OK 2022-08-17T13:33:38.3481689Z 2022-08-17T13:33:38.3481785Z Generating XML reports... 2022-08-17T13:33:38.3515640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133336.xml 2022-08-17T13:33:39.6465843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:39.6466281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:39.6468131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:39.6468521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:39.9905496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:39.9915509Z 2022-08-17T13:33:39.9916091Z Running tests... 2022-08-17T13:33:39.9916506Z ---------------------------------------------------------------------- 2022-08-17T13:33:40.2587237Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17944 2022-08-17T13:33:40.2614870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17945 2022-08-17T13:33:40.2642531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17946 2022-08-17T13:33:41.2457848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:41.2458468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:41.2459764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:41.2460365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:41.2817923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:41.2818310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:41.2821095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:41.2821577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:41.2983031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:41.2983457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:41.2986870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:41.2987253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:41.6692415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:41.6794069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:41.6794930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:41.6795510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:41.6796719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:41.6797436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:41.6802119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:41.6803255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:41.6803937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:41.9689416Z skip: CUDA is not available. (1.977s) 2022-08-17T13:33:41.9689822Z 2022-08-17T13:33:41.9690189Z ---------------------------------------------------------------------- 2022-08-17T13:33:41.9690653Z Ran 1 test in 1.977s 2022-08-17T13:33:41.9690875Z 2022-08-17T13:33:41.9691027Z OK (skipped=1) 2022-08-17T13:33:41.9691152Z 2022-08-17T13:33:41.9691243Z Generating XML reports... 2022-08-17T13:33:41.9725337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133339.xml 2022-08-17T13:33:43.2367084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:43.2367579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:43.2368758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:43.2369337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:43.5882364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:43.5891556Z 2022-08-17T13:33:43.5891698Z Running tests... 2022-08-17T13:33:43.5892336Z ---------------------------------------------------------------------- 2022-08-17T13:33:43.8527662Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17997 2022-08-17T13:33:43.8554867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17998 2022-08-17T13:33:43.8582614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17999 2022-08-17T13:33:44.8438495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:44.8439098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:44.8441637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:44.8442122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:44.8609557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:44.8610216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:44.8613507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:44.8614403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:44.9087426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:44.9087998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:44.9090267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:44.9090799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:45.2624936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:45.2725435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:45.2726292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:45.2727140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:45.2727919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:45.2728832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:45.2832684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:45.3740856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:45.3741498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:45.6628778Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:33:45.6629099Z 2022-08-17T13:33:45.6629579Z ---------------------------------------------------------------------- 2022-08-17T13:33:45.6629843Z Ran 1 test in 2.074s 2022-08-17T13:33:45.6629966Z 2022-08-17T13:33:45.6630044Z OK (skipped=1) 2022-08-17T13:33:45.6630174Z 2022-08-17T13:33:45.6630250Z Generating XML reports... 2022-08-17T13:33:45.6664482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133343.xml 2022-08-17T13:33:46.9298662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:46.9299062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:46.9302216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:46.9302584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:47.2656804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:47.2666861Z 2022-08-17T13:33:47.2666995Z Running tests... 2022-08-17T13:33:47.2667597Z ---------------------------------------------------------------------- 2022-08-17T13:33:47.5267041Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18050 2022-08-17T13:33:47.5294917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18051 2022-08-17T13:33:47.5322857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18052 2022-08-17T13:33:48.5082376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:48.5083048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:48.5084065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:48.5084639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:48.5085691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:48.5086816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:48.5089064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:48.5089675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:48.5401901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:48.5402567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:48.5405065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:48.5405486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:48.9124629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:48.9225179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:48.9225807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:48.9226464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:48.9227022Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:48.9227558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:48.9334223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:48.9334694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:48.9400355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpfr5kdi 2022-08-17T13:33:48.9401098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan1f99d1 2022-08-17T13:33:48.9401914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpfr5kdi/_remote_module_non_scriptable.py 2022-08-17T13:33:48.9402362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan1f99d1/_remote_module_non_scriptable.py 2022-08-17T13:33:49.0238139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:49.0304732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80w_gu0x 2022-08-17T13:33:49.0307008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80w_gu0x/_remote_module_non_scriptable.py 2022-08-17T13:33:49.3371167Z ok (2.070s) 2022-08-17T13:33:49.3371444Z 2022-08-17T13:33:49.3371992Z ---------------------------------------------------------------------- 2022-08-17T13:33:49.3372383Z Ran 1 test in 2.070s 2022-08-17T13:33:49.3372508Z 2022-08-17T13:33:49.3372574Z OK 2022-08-17T13:33:49.3372672Z 2022-08-17T13:33:49.3372770Z Generating XML reports... 2022-08-17T13:33:49.3406869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133347.xml 2022-08-17T13:33:50.6303042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:50.6303439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:50.6306127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:50.6306693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:50.9750280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:50.9760141Z 2022-08-17T13:33:50.9760291Z Running tests... 2022-08-17T13:33:50.9760916Z ---------------------------------------------------------------------- 2022-08-17T13:33:51.2386988Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18103 2022-08-17T13:33:51.2415156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18104 2022-08-17T13:33:51.2444047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18105 2022-08-17T13:33:52.2186478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:52.2186885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:52.2189529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:52.2190054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:52.2320637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:52.2321242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:52.2321821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:52.2322169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:52.2325112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:52.2325756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:52.2326706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:52.2327171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:52.5906819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:52.6002703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:52.6003138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:52.6003769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:52.6004331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:52.6008097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:52.6111349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:52.6111846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:52.7020348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:53.0493648Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:33:53.0493970Z 2022-08-17T13:33:53.0494503Z ---------------------------------------------------------------------- 2022-08-17T13:33:53.0494813Z Ran 1 test in 2.073s 2022-08-17T13:33:53.0494933Z 2022-08-17T13:33:53.0495013Z OK (skipped=1) 2022-08-17T13:33:53.0495128Z 2022-08-17T13:33:53.0495217Z Generating XML reports... 2022-08-17T13:33:53.0529877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133350.xml 2022-08-17T13:33:54.3367611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:54.3368320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:54.3371022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:54.3371920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:54.6794063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:54.6804310Z 2022-08-17T13:33:54.6804655Z Running tests... 2022-08-17T13:33:54.6805320Z ---------------------------------------------------------------------- 2022-08-17T13:33:54.9501627Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18156 2022-08-17T13:33:54.9529902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18157 2022-08-17T13:33:54.9558308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18158 2022-08-17T13:33:55.9321796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:55.9322426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:55.9325387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:55.9326180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:55.9338784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:55.9339372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:55.9343420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:55.9344027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:55.9745789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:55.9746526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:55.9749323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:55.9750013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:56.3417104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:33:56.3518394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:33:56.3518974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:33:56.3519933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:56.3520971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:56.3521517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:33:56.3526627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:33:56.3527470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:33:56.3530745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:33:56.6604250Z skip: Need at least 2 CUDA devices (1.980s) 2022-08-17T13:33:56.6604567Z 2022-08-17T13:33:56.6605098Z ---------------------------------------------------------------------- 2022-08-17T13:33:56.6605389Z Ran 1 test in 1.980s 2022-08-17T13:33:56.6605509Z 2022-08-17T13:33:56.6605585Z OK (skipped=1) 2022-08-17T13:33:56.6605716Z 2022-08-17T13:33:56.6605805Z Generating XML reports... 2022-08-17T13:33:56.6641006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133354.xml 2022-08-17T13:33:57.9681977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:57.9682615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:57.9684435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:57.9684803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:58.3158055Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:58.3167133Z 2022-08-17T13:33:58.3167465Z Running tests... 2022-08-17T13:33:58.3168186Z ---------------------------------------------------------------------- 2022-08-17T13:33:58.3185710Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:33:58.3186201Z 2022-08-17T13:33:58.3186570Z ---------------------------------------------------------------------- 2022-08-17T13:33:58.3187013Z Ran 1 test in 0.002s 2022-08-17T13:33:58.3187227Z 2022-08-17T13:33:58.3187361Z OK (skipped=1) 2022-08-17T13:33:58.3187565Z 2022-08-17T13:33:58.3187719Z Generating XML reports... 2022-08-17T13:33:58.3212760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133358.xml 2022-08-17T13:33:59.4811843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:33:59.4812502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:33:59.4814998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:33:59.4815617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:33:59.8246227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:33:59.8256608Z 2022-08-17T13:33:59.8256905Z Running tests... 2022-08-17T13:33:59.8257568Z ---------------------------------------------------------------------- 2022-08-17T13:33:59.8274892Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:33:59.8275368Z 2022-08-17T13:33:59.8275737Z ---------------------------------------------------------------------- 2022-08-17T13:33:59.8276138Z Ran 1 test in 0.002s 2022-08-17T13:33:59.8276337Z 2022-08-17T13:33:59.8276465Z OK (skipped=1) 2022-08-17T13:33:59.8276663Z 2022-08-17T13:33:59.8276820Z Generating XML reports... 2022-08-17T13:33:59.8302033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133359.xml 2022-08-17T13:34:00.2848286Z Running distributed tests for the gloo backend with file init_method 2022-08-17T13:34:00.2850264Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:34:00.284786] 2022-08-17T13:34:01.5784265Z 2022-08-17T13:34:01.5820837Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-08-17T13:34:01.5849229Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5849643Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5849978Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5850305Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5850646Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5851014Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5851399Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5851771Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5852170Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5852597Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5853119Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5853520Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5853942Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5854347Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5854724Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5855096Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5855461Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5855800Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5856120Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5856471Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5856852Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5857221Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5857579Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5857892Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5858224Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5858542Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5858867Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5859198Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5859502Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5859802Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5860106Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5860418Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5860703Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5861005Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5861323Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5861639Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5861961Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5862300Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5862729Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5863070Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5863421Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5863754Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5864068Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5864412Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5864756Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5865068Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5865413Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5865751Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5866071Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5866378Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5866768Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5867101Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5867401Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5867728Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5868051Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5868373Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5868662Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5868972Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5869287Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5869571Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5869861Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5870165Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5870469Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5870781Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5871122Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5871420Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5871700Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5872004Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5872309Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5872603Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5872925Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5873226Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5873504Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5873798Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5874099Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5874410Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5874708Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5875015Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5875312Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5875610Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5875942Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5876283Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5876614Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5876972Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5877324Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5877675Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5878008Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5878344Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5878683Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5879009Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5879356Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5879711Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5880107Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5880443Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5880792Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5881125Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5881419Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5881713Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5881998Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5882283Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5882563Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5882871Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5883172Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5883455Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5883767Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5884080Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5884405Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5884710Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5885021Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5885352Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5885658Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5886237Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5886562Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5886868Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5887203Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5887536Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5887847Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5888149Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5888437Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5888740Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5889029Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5889328Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5889635Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5889975Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5890378Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5890729Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5891046Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5891361Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5891698Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5892045Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5892396Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5892735Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5893064Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5893392Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5893792Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5894081Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5894383Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5894693Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5895013Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5895353Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5895681Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5896005Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5896328Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5896689Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5897071Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5897514Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5898041Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5898520Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5898988Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5899459Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5899938Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5900404Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5900864Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5901286Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5901671Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5902019Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5902307Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5902614Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5903009Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5903302Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5903630Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5903977Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5904320Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5904684Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5905009Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5905315Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5905618Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5905957Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5906338Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5906646Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5906967Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5907305Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5907641Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5907954Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5908270Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5908586Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5908895Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5909205Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5909530Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5909876Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5910211Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5910504Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5910847Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5911163Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5911491Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5911786Z test_gather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5912054Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5912341Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5912636Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5912913Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5913204Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5913503Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5913798Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5914065Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5914346Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5914638Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5914934Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5915236Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5915523Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5915776Z test_isend (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5916068Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5916380Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5916701Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5917037Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5917388Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5917710Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5918019Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5918355Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5918686Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5919000Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5919323Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5919652Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5920007Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5920311Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5920615Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5920928Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5921276Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5921660Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5922025Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5922373Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5922718Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5923073Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5923411Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5923725Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5924086Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5924430Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5924779Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5925147Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5925529Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5926076Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5926415Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5926712Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5927024Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5927333Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5927620Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5927916Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5928214Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5928514Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5928785Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5929066Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5929352Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5929635Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5929920Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5930208Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5930491Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5930787Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5931069Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5931353Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5931627Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5931913Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5932209Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5932499Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5932788Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5933080Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5933353Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5933719Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5934048Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5934395Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5934716Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5935025Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5935338Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5935657Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5935975Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5936284Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5936589Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5936927Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5937245Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5937551Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5937893Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5938201Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5938497Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5938808Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5939153Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:01.5939500Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-08-17T13:34:02.5288593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:02.5289095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:02.5291364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:02.5291967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:02.8695064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:02.8703804Z 2022-08-17T13:34:02.8703928Z Running tests... 2022-08-17T13:34:02.8704459Z ---------------------------------------------------------------------- 2022-08-17T13:34:03.1384690Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18239 2022-08-17T13:34:03.1413249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18240 2022-08-17T13:34:03.1441682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18241 2022-08-17T13:34:04.1334132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:04.1334596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:04.1337035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:04.1337522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:04.1467278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:04.1467942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:04.1470969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:04.1471635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:04.1551780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:04.1552490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:04.1556441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:04.1557083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:04.5119537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:04.5218769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:04.5219513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:04.5220651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:04.5221225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:04.5221783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:04.5327199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:04.5327785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:04.6234311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:04.9490258Z skip: Need at least 2 CUDA devices (2.078s) 2022-08-17T13:34:04.9490560Z 2022-08-17T13:34:04.9491045Z ---------------------------------------------------------------------- 2022-08-17T13:34:04.9491433Z Ran 1 test in 2.079s 2022-08-17T13:34:04.9491619Z 2022-08-17T13:34:04.9491737Z OK (skipped=1) 2022-08-17T13:34:04.9491935Z 2022-08-17T13:34:04.9492076Z Generating XML reports... 2022-08-17T13:34:04.9527287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133402.xml 2022-08-17T13:34:06.2696586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:06.2697032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:06.2705333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:06.2705722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:06.6114271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:06.6124269Z 2022-08-17T13:34:06.6124388Z Running tests... 2022-08-17T13:34:06.6124962Z ---------------------------------------------------------------------- 2022-08-17T13:34:06.6161547Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-08-17T13:34:06.6162017Z 2022-08-17T13:34:06.6162442Z ---------------------------------------------------------------------- 2022-08-17T13:34:06.6162868Z Ran 1 test in 0.004s 2022-08-17T13:34:06.6163060Z 2022-08-17T13:34:06.6163186Z OK (skipped=1) 2022-08-17T13:34:06.6163375Z 2022-08-17T13:34:06.6163534Z Generating XML reports... 2022-08-17T13:34:06.6188951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133406.xml 2022-08-17T13:34:07.7584455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:07.7584864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:07.7587505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:07.7588199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:08.1039948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:08.1049907Z 2022-08-17T13:34:08.1050051Z Running tests... 2022-08-17T13:34:08.1050717Z ---------------------------------------------------------------------- 2022-08-17T13:34:08.3712601Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18302 2022-08-17T13:34:08.3740587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18303 2022-08-17T13:34:08.3768541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18304 2022-08-17T13:34:09.3975678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:09.3976106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:09.3977145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:09.3977730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:09.4172189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:09.4172571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:09.4175424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:09.4175858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:09.4259390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:09.4259936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:09.4263889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:09.4264433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:09.7931775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:09.7932286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:09.7932758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:09.7933422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:09.7934016Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:09.7934625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:09.8040160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:09.8040805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:09.8944068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:10.1818916Z ok (2.077s) 2022-08-17T13:34:10.1819108Z 2022-08-17T13:34:10.1819482Z ---------------------------------------------------------------------- 2022-08-17T13:34:10.1819770Z Ran 1 test in 2.077s 2022-08-17T13:34:10.1819892Z 2022-08-17T13:34:10.1819957Z OK 2022-08-17T13:34:10.1820085Z 2022-08-17T13:34:10.1820202Z Generating XML reports... 2022-08-17T13:34:10.1854855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133408.xml 2022-08-17T13:34:11.4979373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:11.4980050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:11.4982407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:11.4982864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:11.8369022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:11.8378817Z 2022-08-17T13:34:11.8378921Z Running tests... 2022-08-17T13:34:11.8379377Z ---------------------------------------------------------------------- 2022-08-17T13:34:12.1012700Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18355 2022-08-17T13:34:12.1041737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18356 2022-08-17T13:34:12.1070171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18357 2022-08-17T13:34:13.0860363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:13.0860795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:13.0864065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:13.0864453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:13.0965141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:13.0965760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:13.0968407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:13.0969046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:13.0969893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:13.0970516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:13.0973562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:13.0974011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:13.4766252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:13.4766893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:13.4767293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:13.4767945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:13.4768514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:13.4769050Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:13.4872291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:13.5779380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:13.5780049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:13.9118701Z skip: CUDA is not available. (2.074s) 2022-08-17T13:34:13.9119031Z 2022-08-17T13:34:13.9119527Z ---------------------------------------------------------------------- 2022-08-17T13:34:13.9120028Z Ran 1 test in 2.074s 2022-08-17T13:34:13.9120197Z 2022-08-17T13:34:13.9120277Z OK (skipped=1) 2022-08-17T13:34:13.9120388Z 2022-08-17T13:34:13.9120704Z Generating XML reports... 2022-08-17T13:34:13.9155685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133411.xml 2022-08-17T13:34:15.2092037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:15.2092454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:15.2095247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:15.2095748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:15.5554633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:15.5564067Z 2022-08-17T13:34:15.5564196Z Running tests... 2022-08-17T13:34:15.5564848Z ---------------------------------------------------------------------- 2022-08-17T13:34:15.8212555Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18408 2022-08-17T13:34:15.8241222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18409 2022-08-17T13:34:15.8269163Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18410 2022-08-17T13:34:16.8075511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:16.8076212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:16.8106141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:16.8106821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:16.8124826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:16.8125296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:16.8129101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:16.8129465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:16.8440884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:16.8441265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:16.8444759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:16.8445117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:17.2118313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:17.2119254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:17.2120014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:17.2120657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:17.2121236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:17.2121789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:17.2127053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:17.2128285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:17.2129279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:17.2200423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu3s3mv8 2022-08-17T13:34:17.2201168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wl3bsd0 2022-08-17T13:34:17.2202084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxjsssmb 2022-08-17T13:34:17.2202824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu3s3mv8/_remote_module_non_scriptable.py 2022-08-17T13:34:17.2203428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wl3bsd0/_remote_module_non_scriptable.py 2022-08-17T13:34:17.2204073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxjsssmb/_remote_module_non_scriptable.py 2022-08-17T13:34:17.2377828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:17.2378446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:17.2378813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:17.6318151Z ok (2.075s) 2022-08-17T13:34:17.6318343Z 2022-08-17T13:34:17.6318704Z ---------------------------------------------------------------------- 2022-08-17T13:34:17.6318974Z Ran 1 test in 2.075s 2022-08-17T13:34:17.6319097Z 2022-08-17T13:34:17.6319148Z OK 2022-08-17T13:34:17.6319245Z 2022-08-17T13:34:17.6319562Z Generating XML reports... 2022-08-17T13:34:17.6355726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133415.xml 2022-08-17T13:34:18.9250930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:18.9251327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:18.9254014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:18.9254548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:19.2620974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:19.2631714Z 2022-08-17T13:34:19.2632148Z Running tests... 2022-08-17T13:34:19.2632557Z ---------------------------------------------------------------------- 2022-08-17T13:34:19.5232047Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18461 2022-08-17T13:34:19.5261018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18462 2022-08-17T13:34:19.5288734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18463 2022-08-17T13:34:20.5305303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:20.5305718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:20.5308423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:20.5309014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:20.5718994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:20.5721671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:20.5722237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:20.5722611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:20.5884538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:20.5885098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:20.5888505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:20.5889367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:20.9492052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:20.9492733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:20.9493376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:20.9494240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:20.9494803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:20.9495336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:20.9599275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:20.9599882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:20.9669488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo0x6jco 2022-08-17T13:34:20.9670200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp471xpaux 2022-08-17T13:34:20.9671979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo0x6jco/_remote_module_non_scriptable.py 2022-08-17T13:34:20.9672507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp471xpaux/_remote_module_non_scriptable.py 2022-08-17T13:34:21.0504116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:21.0576156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu0dj52v 2022-08-17T13:34:21.0577876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu0dj52v/_remote_module_non_scriptable.py 2022-08-17T13:34:21.0734958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:21.0735648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:21.0736101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:34:21.5342394Z ok (2.271s) 2022-08-17T13:34:21.5342781Z 2022-08-17T13:34:21.5343283Z ---------------------------------------------------------------------- 2022-08-17T13:34:21.5343704Z Ran 1 test in 2.271s 2022-08-17T13:34:21.5343895Z 2022-08-17T13:34:21.5343963Z OK 2022-08-17T13:34:21.5344385Z 2022-08-17T13:34:21.5344531Z Generating XML reports... 2022-08-17T13:34:21.5381208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133419.xml 2022-08-17T13:34:22.8387718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:22.8388134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:22.8390624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:22.8391047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:23.1770805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:23.1779982Z 2022-08-17T13:34:23.1780578Z Running tests... 2022-08-17T13:34:23.1780994Z ---------------------------------------------------------------------- 2022-08-17T13:34:23.4392658Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18514 2022-08-17T13:34:23.4420518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18515 2022-08-17T13:34:23.4448552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18516 2022-08-17T13:34:24.4136680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:24.4137300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:24.4140302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:24.4140968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:24.4161145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:24.4161769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:24.4166179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:24.4166802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:24.4406215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:24.4406853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:24.4409945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:24.4410598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:24.7988502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:24.8016899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:24.8017504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:24.8018160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:24.8018717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:24.8089493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:24.8124994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:24.8125574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:24.9101168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:25.2496809Z skip: CUDA is not available. (2.071s) 2022-08-17T13:34:25.2497697Z 2022-08-17T13:34:25.2498282Z ---------------------------------------------------------------------- 2022-08-17T13:34:25.2498753Z Ran 1 test in 2.072s 2022-08-17T13:34:25.2498886Z 2022-08-17T13:34:25.2498967Z OK (skipped=1) 2022-08-17T13:34:25.2499091Z 2022-08-17T13:34:25.2499181Z Generating XML reports... 2022-08-17T13:34:25.2534204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133423.xml 2022-08-17T13:34:26.5316878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:26.5317343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:26.5319931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:26.5320302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:26.8742718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:26.8752876Z 2022-08-17T13:34:26.8753006Z Running tests... 2022-08-17T13:34:26.8753620Z ---------------------------------------------------------------------- 2022-08-17T13:34:27.1384736Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18567 2022-08-17T13:34:27.1412588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18568 2022-08-17T13:34:27.1440257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18569 2022-08-17T13:34:28.1179607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:28.1180259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:28.1183022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:28.1183641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:28.1191885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:28.1192533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:28.1196734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:28.1197645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:28.1218757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:28.1219394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:28.1223118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:28.1223538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:28.4879951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:28.4980759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:28.4981901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:28.4982723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:28.4983486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:28.4984030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:28.4988889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:28.4989645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:28.4990508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:28.8490168Z skip: CUDA is not available. (1.973s) 2022-08-17T13:34:28.8490487Z 2022-08-17T13:34:28.8490861Z ---------------------------------------------------------------------- 2022-08-17T13:34:28.8491146Z Ran 1 test in 1.974s 2022-08-17T13:34:28.8491267Z 2022-08-17T13:34:28.8491346Z OK (skipped=1) 2022-08-17T13:34:28.8491479Z 2022-08-17T13:34:28.8491579Z Generating XML reports... 2022-08-17T13:34:28.8526006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133426.xml 2022-08-17T13:34:30.1494573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:30.1495039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:30.1496575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:30.1496959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:30.4933712Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:30.4942822Z 2022-08-17T13:34:30.4943270Z Running tests... 2022-08-17T13:34:30.4943690Z ---------------------------------------------------------------------- 2022-08-17T13:34:30.7604452Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18620 2022-08-17T13:34:30.7633122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18621 2022-08-17T13:34:30.7660741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18622 2022-08-17T13:34:31.7594460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:31.7595111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:31.7597435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:31.7598095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:31.7913315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:31.7913975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:31.7916357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:31.7916939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:31.7919391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:31.7919767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:31.7924458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:31.7924855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:32.1695866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:32.1696334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:32.1696935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:32.1697915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:32.1698573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:32.1699185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:32.1705516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:32.1706137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:32.2709784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:32.5707451Z skip: CUDA is not available. (2.076s) 2022-08-17T13:34:32.5707771Z 2022-08-17T13:34:32.5708294Z ---------------------------------------------------------------------- 2022-08-17T13:34:32.5708629Z Ran 1 test in 2.076s 2022-08-17T13:34:32.5708753Z 2022-08-17T13:34:32.5708832Z OK (skipped=1) 2022-08-17T13:34:32.5708948Z 2022-08-17T13:34:32.5709023Z Generating XML reports... 2022-08-17T13:34:32.5746442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133430.xml 2022-08-17T13:34:33.8494308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:33.8495228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:33.8497310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:33.8497923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:34.2039907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:34.2050158Z 2022-08-17T13:34:34.2050483Z Running tests... 2022-08-17T13:34:34.2051153Z ---------------------------------------------------------------------- 2022-08-17T13:34:34.4700242Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18673 2022-08-17T13:34:34.4727287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18674 2022-08-17T13:34:34.4754758Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18675 2022-08-17T13:34:35.4565410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:35.4566370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:35.4568833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:35.4569447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:35.4610278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:35.4610915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:35.4615190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:35.4615871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:35.4737889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:35.4738318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:35.4741505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:35.4742009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:35.8388921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:35.8489991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:35.8490398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:35.8491034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:35.8491610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:35.8492159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:35.8497027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:35.8498375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:35.8499052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:36.1801280Z skip: CUDA is not available. (1.975s) 2022-08-17T13:34:36.1801593Z 2022-08-17T13:34:36.1802094Z ---------------------------------------------------------------------- 2022-08-17T13:34:36.1802563Z Ran 1 test in 1.975s 2022-08-17T13:34:36.1802790Z 2022-08-17T13:34:36.1803161Z OK (skipped=1) 2022-08-17T13:34:36.1803370Z 2022-08-17T13:34:36.1803533Z Generating XML reports... 2022-08-17T13:34:36.1837842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133434.xml 2022-08-17T13:34:37.4526797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:37.4527213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:37.4529770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:37.4530284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:37.7937884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:37.7947500Z 2022-08-17T13:34:37.7947630Z Running tests... 2022-08-17T13:34:37.7948246Z ---------------------------------------------------------------------- 2022-08-17T13:34:38.0601848Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18726 2022-08-17T13:34:38.0630187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18727 2022-08-17T13:34:38.0658193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18728 2022-08-17T13:34:39.0479561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:39.0487154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:39.0489853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:39.0490533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:39.0520388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:39.0520784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:39.0524420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:39.0524812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:39.0576500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:39.0577132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:39.0581467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:39.0582073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:39.4410533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:39.4511886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:39.4512360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:39.4513053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:39.4513666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:39.4514570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:39.4520311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:39.4521551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:39.4522126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:39.7704786Z skip: CUDA is not available. (1.975s) 2022-08-17T13:34:39.7705086Z 2022-08-17T13:34:39.7705545Z ---------------------------------------------------------------------- 2022-08-17T13:34:39.7705962Z Ran 1 test in 1.976s 2022-08-17T13:34:39.7706148Z 2022-08-17T13:34:39.7706287Z OK (skipped=1) 2022-08-17T13:34:39.7706477Z 2022-08-17T13:34:39.7706620Z Generating XML reports... 2022-08-17T13:34:39.7741920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133437.xml 2022-08-17T13:34:41.0390939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:41.0391638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:41.0394429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:41.0395062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:41.3832630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:41.3843269Z 2022-08-17T13:34:41.3843803Z Running tests... 2022-08-17T13:34:41.3844502Z ---------------------------------------------------------------------- 2022-08-17T13:34:41.6513251Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18779 2022-08-17T13:34:41.6542051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18780 2022-08-17T13:34:41.6571806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18781 2022-08-17T13:34:42.6306111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:42.6306718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:42.6308688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:42.6309347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:42.6453647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:42.6454331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:42.6456407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:42.6457039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:42.6735964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:42.6736620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:42.6739532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:42.6740142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:43.0311192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:43.0412523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:43.0413550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:43.0414333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:43.0415218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:43.0416133Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:43.0520254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:43.1429102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:43.1429944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:43.4620359Z skip: CUDA is not available. (2.077s) 2022-08-17T13:34:43.4620659Z 2022-08-17T13:34:43.4621110Z ---------------------------------------------------------------------- 2022-08-17T13:34:43.4621591Z Ran 1 test in 2.078s 2022-08-17T13:34:43.4621807Z 2022-08-17T13:34:43.4621881Z OK (skipped=1) 2022-08-17T13:34:43.4621997Z 2022-08-17T13:34:43.4622088Z Generating XML reports... 2022-08-17T13:34:43.4657810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133441.xml 2022-08-17T13:34:44.7722808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:44.7723530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:44.7726628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:44.7727512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:45.1186717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:45.1197026Z 2022-08-17T13:34:45.1197369Z Running tests... 2022-08-17T13:34:45.1198008Z ---------------------------------------------------------------------- 2022-08-17T13:34:45.3915002Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18832 2022-08-17T13:34:45.3943461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18833 2022-08-17T13:34:45.3975185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18834 2022-08-17T13:34:46.3724296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:46.3724952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:46.3727666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:46.3728351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:46.3764352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:46.3764970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:46.3768912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:46.3769573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:46.4211771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:46.4212454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:46.4216046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:46.4216702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:46.7842702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:46.7943072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:46.7943662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:46.7944607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:46.7945736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:46.7946699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:46.8049172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:46.8955935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:46.8956581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:47.2021669Z skip: CUDA is not available. (2.082s) 2022-08-17T13:34:47.2022276Z 2022-08-17T13:34:47.2023045Z ---------------------------------------------------------------------- 2022-08-17T13:34:47.2023541Z Ran 1 test in 2.082s 2022-08-17T13:34:47.2023759Z 2022-08-17T13:34:47.2023871Z OK (skipped=1) 2022-08-17T13:34:47.2023991Z 2022-08-17T13:34:47.2024082Z Generating XML reports... 2022-08-17T13:34:47.2057825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133445.xml 2022-08-17T13:34:48.4871745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:48.4872154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:48.4874101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:48.4874743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:48.8272120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:48.8281778Z 2022-08-17T13:34:48.8281923Z Running tests... 2022-08-17T13:34:48.8282559Z ---------------------------------------------------------------------- 2022-08-17T13:34:49.0904735Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18885 2022-08-17T13:34:49.0933868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18886 2022-08-17T13:34:49.0962392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18887 2022-08-17T13:34:50.0478571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:50.0479242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:50.0481339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:50.0481987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:50.0483549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:50.0484190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:50.0484980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:50.0485619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:50.0943180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:50.0943803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:50.0945465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:50.0946096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:50.4539595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:50.4579152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:50.4579828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:50.4580474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:50.4581042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:50.4640158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:50.4687626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:50.4688353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:50.5657420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:50.8123724Z skip: Need at least 3 CUDA devices (1.984s) 2022-08-17T13:34:50.8124045Z 2022-08-17T13:34:50.8124530Z ---------------------------------------------------------------------- 2022-08-17T13:34:50.8124807Z Ran 1 test in 1.984s 2022-08-17T13:34:50.8124927Z 2022-08-17T13:34:50.8125221Z OK (skipped=1) 2022-08-17T13:34:50.8125412Z 2022-08-17T13:34:50.8125554Z Generating XML reports... 2022-08-17T13:34:50.8160268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133448.xml 2022-08-17T13:34:52.0675946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:52.0676347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:52.0679348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:52.0680066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:52.4062265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:52.4072021Z 2022-08-17T13:34:52.4072177Z Running tests... 2022-08-17T13:34:52.4072851Z ---------------------------------------------------------------------- 2022-08-17T13:34:52.6745332Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18938 2022-08-17T13:34:52.6773879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18939 2022-08-17T13:34:52.6803289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18940 2022-08-17T13:34:53.6479115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:53.6479809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:53.6483195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:53.6483777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:53.6535650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:53.6536045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:53.6574528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:53.6575177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:53.6840952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:53.6841592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:53.6844788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:53.6845751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:54.0515842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:54.0618070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:54.0618570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:54.0618965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:54.0619501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:54.0620038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:54.0625002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:54.0625872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:54.1627732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:54.5853024Z ok (2.178s) 2022-08-17T13:34:54.5853270Z 2022-08-17T13:34:54.5853608Z ---------------------------------------------------------------------- 2022-08-17T13:34:54.5853875Z Ran 1 test in 2.178s 2022-08-17T13:34:54.5853998Z 2022-08-17T13:34:54.5854061Z OK 2022-08-17T13:34:54.5854156Z 2022-08-17T13:34:54.5854243Z Generating XML reports... 2022-08-17T13:34:54.5892934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133452.xml 2022-08-17T13:34:55.8950462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:55.8950869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:55.8953287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:55.8953678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:56.2375136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:56.2385066Z 2022-08-17T13:34:56.2385210Z Running tests... 2022-08-17T13:34:56.2385837Z ---------------------------------------------------------------------- 2022-08-17T13:34:56.5059964Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18991 2022-08-17T13:34:56.5087659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18992 2022-08-17T13:34:56.5115963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18993 2022-08-17T13:34:57.4927896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:57.4928529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:57.4932010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:57.4932734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:57.4933540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:57.4934094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:57.4938270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:57.4938877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:57.4949270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:57.4949869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:57.4953916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:57.4954318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:57.8642210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:34:57.8743190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:34:57.8743730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:34:57.8744397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:57.8744972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:57.8745510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:34:57.8851637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:34:57.8852344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:34:57.9755920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:34:58.3164000Z skip: CUDA is not available. (2.078s) 2022-08-17T13:34:58.3164333Z 2022-08-17T13:34:58.3164821Z ---------------------------------------------------------------------- 2022-08-17T13:34:58.3165243Z Ran 1 test in 2.078s 2022-08-17T13:34:58.3165429Z 2022-08-17T13:34:58.3165532Z OK (skipped=1) 2022-08-17T13:34:58.3165738Z 2022-08-17T13:34:58.3166064Z Generating XML reports... 2022-08-17T13:34:58.3202942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133456.xml 2022-08-17T13:34:59.6269239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:34:59.6269631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:34:59.6271882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:34:59.6272258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:34:59.9859119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:34:59.9868796Z 2022-08-17T13:34:59.9868949Z Running tests... 2022-08-17T13:34:59.9869393Z ---------------------------------------------------------------------- 2022-08-17T13:35:00.2546774Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19044 2022-08-17T13:35:00.2574761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19045 2022-08-17T13:35:00.2603029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19046 2022-08-17T13:35:01.2329151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:01.2329810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:01.2330646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:01.2331305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:01.2422951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:01.2423961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:01.2427768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:01.2428373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:01.2694419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:01.2695053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:01.2697931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:01.2698583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:01.6352859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:01.6454942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:01.6455390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:01.6456172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:01.6457271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:01.6457953Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:01.6463308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:01.6464203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:01.6464815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:01.9649566Z skip: Need at least 3 CUDA devices (1.978s) 2022-08-17T13:35:01.9649911Z 2022-08-17T13:35:01.9650379Z ---------------------------------------------------------------------- 2022-08-17T13:35:01.9650629Z Ran 1 test in 1.978s 2022-08-17T13:35:01.9650751Z 2022-08-17T13:35:01.9650828Z OK (skipped=1) 2022-08-17T13:35:01.9650940Z 2022-08-17T13:35:01.9651047Z Generating XML reports... 2022-08-17T13:35:01.9686022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133459.xml 2022-08-17T13:35:03.2385057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:03.2385439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:03.2387964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:03.2388340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:03.5852306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:03.5862495Z 2022-08-17T13:35:03.5862622Z Running tests... 2022-08-17T13:35:03.5863163Z ---------------------------------------------------------------------- 2022-08-17T13:35:03.5882256Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-08-17T13:35:03.5882559Z 2022-08-17T13:35:03.5882894Z ---------------------------------------------------------------------- 2022-08-17T13:35:03.5883143Z Ran 1 test in 0.002s 2022-08-17T13:35:03.5883262Z 2022-08-17T13:35:03.5883345Z OK (skipped=1) 2022-08-17T13:35:03.5883460Z 2022-08-17T13:35:03.5883550Z Generating XML reports... 2022-08-17T13:35:03.5908861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133503.xml 2022-08-17T13:35:04.7582912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:04.7583624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:04.7585957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:04.7586500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:05.1055231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:05.1064726Z 2022-08-17T13:35:05.1064874Z Running tests... 2022-08-17T13:35:05.1065508Z ---------------------------------------------------------------------- 2022-08-17T13:35:05.1079812Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-08-17T13:35:05.3709420Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19107 2022-08-17T13:35:05.3738462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19108 2022-08-17T13:35:05.3767970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19109 2022-08-17T13:35:06.3606081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:06.3607098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:06.3610195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:06.3610603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:06.3667672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:06.3668070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:06.3672587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:06.3673238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:06.3707940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:06.3708601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:06.3711884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:06.3712555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:06.7489811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:06.7490469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:06.7490872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:06.7491561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:06.7492208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:06.7492758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:06.7498269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:06.7498851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:06.7499787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:06.7574303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0g549lv 2022-08-17T13:35:06.7575381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5k4w8h2 2022-08-17T13:35:06.7576416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0g549lv/_remote_module_non_scriptable.py 2022-08-17T13:35:06.7577166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjyj0vnd 2022-08-17T13:35:06.7577762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5k4w8h2/_remote_module_non_scriptable.py 2022-08-17T13:35:06.7579100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjyj0vnd/_remote_module_non_scriptable.py 2022-08-17T13:35:07.0815782Z ok (1.975s) 2022-08-17T13:35:07.0816032Z 2022-08-17T13:35:07.0816557Z ---------------------------------------------------------------------- 2022-08-17T13:35:07.0816973Z Ran 1 test in 1.975s 2022-08-17T13:35:07.0817082Z 2022-08-17T13:35:07.0817145Z OK 2022-08-17T13:35:07.0817240Z 2022-08-17T13:35:07.0817337Z Generating XML reports... 2022-08-17T13:35:07.0851282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133505.xml 2022-08-17T13:35:08.3992569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:08.3992961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:08.3995539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:08.3995931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:08.7495710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:08.7505782Z 2022-08-17T13:35:08.7505870Z Running tests... 2022-08-17T13:35:08.7506768Z ---------------------------------------------------------------------- 2022-08-17T13:35:08.7525686Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:35:09.0241535Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19160 2022-08-17T13:35:09.0271859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19161 2022-08-17T13:35:09.0302692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19162 2022-08-17T13:35:10.0065861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:10.0066554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:10.0068982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:10.0069607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:10.0378053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:10.0378682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:10.0381479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:10.0381963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:10.0438552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:10.0438945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:10.0442287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:10.0442652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:10.4189996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:10.4190661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:10.4191300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:10.4192149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:10.4192715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:10.4193250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:10.4296118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:10.4370701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6k795ba 2022-08-17T13:35:10.4372605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6k795ba/_remote_module_non_scriptable.py 2022-08-17T13:35:10.5202573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:10.5203045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:10.5280260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk90cptq6 2022-08-17T13:35:10.5281159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgzxveil 2022-08-17T13:35:10.5281963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk90cptq6/_remote_module_non_scriptable.py 2022-08-17T13:35:10.5282631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgzxveil/_remote_module_non_scriptable.py 2022-08-17T13:35:10.5480470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:10.5481112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:10.5481506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:10.8351319Z ok (2.084s) 2022-08-17T13:35:10.8351599Z 2022-08-17T13:35:10.8352139Z ---------------------------------------------------------------------- 2022-08-17T13:35:10.8352398Z Ran 1 test in 2.084s 2022-08-17T13:35:10.8352523Z 2022-08-17T13:35:10.8352588Z OK 2022-08-17T13:35:10.8352686Z 2022-08-17T13:35:10.8352785Z Generating XML reports... 2022-08-17T13:35:10.8390807Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133508.xml 2022-08-17T13:35:12.1274645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:12.1275024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:12.1277461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:12.1277838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:12.4730748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:12.4741186Z 2022-08-17T13:35:12.4741287Z Running tests... 2022-08-17T13:35:12.4741747Z ---------------------------------------------------------------------- 2022-08-17T13:35:12.4761303Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-08-17T13:35:12.7398712Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19213 2022-08-17T13:35:12.7426976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19214 2022-08-17T13:35:12.7455083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19215 2022-08-17T13:35:13.7406976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:13.7407599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:13.7410607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:13.7411108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:13.7502795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:13.7503172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:13.7506829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:13.7507210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:13.7535063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:13.7535417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:13.7539206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:13.7539597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:14.1136526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:14.1237993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:14.1238533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:14.1239194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:14.1239762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:14.1240318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:14.1345288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:14.1418327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxu7f0yb 2022-08-17T13:35:14.1420507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxu7f0yb/_remote_module_non_scriptable.py 2022-08-17T13:35:14.2250768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:14.2251339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:14.2326288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk0fxjoo 2022-08-17T13:35:14.2326982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92it8ki5 2022-08-17T13:35:14.2327971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk0fxjoo/_remote_module_non_scriptable.py 2022-08-17T13:35:14.2328467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92it8ki5/_remote_module_non_scriptable.py 2022-08-17T13:35:14.2542244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:14.2542989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:14.2543567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:35:14.5505898Z ok (2.076s) 2022-08-17T13:35:14.5506155Z 2022-08-17T13:35:14.5506672Z ---------------------------------------------------------------------- 2022-08-17T13:35:14.5507134Z Ran 1 test in 2.076s 2022-08-17T13:35:14.5507351Z 2022-08-17T13:35:14.5507445Z OK 2022-08-17T13:35:14.5507626Z 2022-08-17T13:35:14.5507791Z Generating XML reports... 2022-08-17T13:35:14.5546053Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133512.xml 2022-08-17T13:35:15.8727475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:15.8728198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:15.8730178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:15.8730576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:16.2190382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:16.2200428Z 2022-08-17T13:35:16.2200579Z Running tests... 2022-08-17T13:35:16.2201068Z ---------------------------------------------------------------------- 2022-08-17T13:35:16.2215908Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-08-17T13:35:16.4964918Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19266 2022-08-17T13:35:16.4993786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19267 2022-08-17T13:35:16.5022063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19268 2022-08-17T13:35:17.5099408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:17.5100094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:17.5103044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:17.5103730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:17.5204169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:17.5204865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:17.5208878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:17.5209529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:17.5404833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:17.5405504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:17.5407513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:17.5408172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:17.9096713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:17.9198403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:17.9199135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:17.9199797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:17.9200367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:17.9200927Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:17.9306139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:17.9380267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqz6q850 2022-08-17T13:35:17.9382195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqz6q850/_remote_module_non_scriptable.py 2022-08-17T13:35:18.0211393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:18.0211922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:18.0288273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8g5yew3 2022-08-17T13:35:18.0290718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8g5yew3/_remote_module_non_scriptable.py 2022-08-17T13:35:18.0291365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4up9hkn 2022-08-17T13:35:18.0293706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4up9hkn/_remote_module_non_scriptable.py 2022-08-17T13:35:18.4072607Z ok (2.187s) 2022-08-17T13:35:18.4072775Z 2022-08-17T13:35:18.4073137Z ---------------------------------------------------------------------- 2022-08-17T13:35:18.4073583Z Ran 1 test in 2.187s 2022-08-17T13:35:18.4073808Z 2022-08-17T13:35:18.4074257Z OK 2022-08-17T13:35:18.4074421Z 2022-08-17T13:35:18.4074602Z Generating XML reports... 2022-08-17T13:35:18.4112739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133516.xml 2022-08-17T13:35:19.7104362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:19.7104762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:19.7107710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:19.7108103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:20.0510447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:20.0520106Z 2022-08-17T13:35:20.0520203Z Running tests... 2022-08-17T13:35:20.0521183Z ---------------------------------------------------------------------- 2022-08-17T13:35:20.3154965Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19319 2022-08-17T13:35:20.3185049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19320 2022-08-17T13:35:20.3213719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19321 2022-08-17T13:35:21.3105012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:21.3105679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:21.3107040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:21.3107696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:21.3149993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:21.3150384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:21.3154518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:21.3155093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:21.3315256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:21.3315801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:21.3318777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:21.3319328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:21.6930020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:21.7030326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:21.7031001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:21.7031973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:21.7032522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:21.7033083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:21.7137761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:21.7138589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:21.7142769Z STAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.7143407Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8042581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:21.8046817Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8055432Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8056865Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8057373Z 2022-08-17T13:35:21.8073349Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8074027Z STAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8074675Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8079814Z STAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8080683Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8081214Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8090995Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8091951Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:21.8092243Z 2022-08-17T13:35:21.8096752Z STAGE:2022-08-17 13:35:21 19320:19320 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8097608Z STAGE:2022-08-17 13:35:21 19319:19319 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:21.8098232Z STAGE:2022-08-17 13:35:21 19321:19321 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:22.2263765Z ok (2.174s) 2022-08-17T13:35:22.2264326Z 2022-08-17T13:35:22.2265121Z ---------------------------------------------------------------------- 2022-08-17T13:35:22.2265562Z Ran 1 test in 2.174s 2022-08-17T13:35:22.2265774Z 2022-08-17T13:35:22.2265908Z OK 2022-08-17T13:35:22.2266057Z 2022-08-17T13:35:22.2266206Z Generating XML reports... 2022-08-17T13:35:22.2302521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133520.xml 2022-08-17T13:35:23.5302132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:23.5302663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:23.5304889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:23.5305535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:23.8764593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:23.8775202Z 2022-08-17T13:35:23.8775333Z Running tests... 2022-08-17T13:35:23.8775794Z ---------------------------------------------------------------------- 2022-08-17T13:35:24.1462880Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19375 2022-08-17T13:35:24.1492500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19376 2022-08-17T13:35:24.1521840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19377 2022-08-17T13:35:25.1407442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:25.1408093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:25.1410790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:25.1411368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:25.1503508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:25.1504192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:25.1507031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:25.1507428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:25.1591126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:25.1591784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:25.1595161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:25.1595856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:25.5186118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:25.5186586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:25.5186958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:25.5187599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:25.5188158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:25.5188721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:25.5293485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:25.5294130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:25.5298718Z STAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.5299462Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6198417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:25.6202183Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6212145Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6212947Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6213969Z STAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6223926Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6224879Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6225188Z 2022-08-17T13:35:25.6231891Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6232449Z 2022-08-17T13:35:25.6232936Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6242472Z STAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6243119Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6243732Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:25.6250977Z STAGE:2022-08-17 13:35:25 19375:19375 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6251988Z STAGE:2022-08-17 13:35:25 19377:19377 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:25.6290945Z STAGE:2022-08-17 13:35:25 19376:19376 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:26.0574460Z ok (2.179s) 2022-08-17T13:35:26.0574712Z 2022-08-17T13:35:26.0575131Z ---------------------------------------------------------------------- 2022-08-17T13:35:26.0575413Z Ran 1 test in 2.180s 2022-08-17T13:35:26.0575537Z 2022-08-17T13:35:26.0575588Z OK 2022-08-17T13:35:26.0575689Z 2022-08-17T13:35:26.0575786Z Generating XML reports... 2022-08-17T13:35:26.0610586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133523.xml 2022-08-17T13:35:27.3706628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:27.3707271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:27.3709558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:27.3710166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:27.7158215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:27.7169294Z 2022-08-17T13:35:27.7169648Z Running tests... 2022-08-17T13:35:27.7170299Z ---------------------------------------------------------------------- 2022-08-17T13:35:27.9792650Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19431 2022-08-17T13:35:27.9821535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19432 2022-08-17T13:35:27.9850546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19433 2022-08-17T13:35:28.9682261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:28.9682665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:28.9685323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:28.9686046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:28.9695827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:28.9696462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:28.9699964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:28.9703573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:28.9704584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:28.9705233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:28.9708451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:28.9709148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:29.3462118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:29.3462659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:29.3463086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:29.3463998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:29.3465100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:29.3466239Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:29.3568923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:29.4474124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:29.4474648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:29.4682166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:35:29.4783737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:35:29.4784653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:29.4785331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:35:29.4786216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:29.4787052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:29.4788102Z STAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4789003Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4789316Z 2022-08-17T13:35:29.4797568Z STAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4798349Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4798662Z 2022-08-17T13:35:29.4807193Z STAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4808362Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4808886Z 2022-08-17T13:35:29.4813354Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4814239Z STAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4814549Z 2022-08-17T13:35:29.4820298Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4823770Z STAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:29.4824117Z 2022-08-17T13:35:29.4829813Z STAGE:2022-08-17 13:35:29 19432:19432 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4830472Z STAGE:2022-08-17 13:35:29 19431:19431 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:29 19433:19433 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:29.4830774Z 2022-08-17T13:35:29.8900437Z ok (2.173s) 2022-08-17T13:35:29.8900643Z 2022-08-17T13:35:29.8900984Z ---------------------------------------------------------------------- 2022-08-17T13:35:29.8901262Z Ran 1 test in 2.173s 2022-08-17T13:35:29.8901386Z 2022-08-17T13:35:29.8901452Z OK 2022-08-17T13:35:29.8901550Z 2022-08-17T13:35:29.8901831Z Generating XML reports... 2022-08-17T13:35:29.8936705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133527.xml 2022-08-17T13:35:31.1811109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:31.1811727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:31.1814077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:31.1814697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:31.5277021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:31.5287234Z 2022-08-17T13:35:31.5287588Z Running tests... 2022-08-17T13:35:31.5288273Z ---------------------------------------------------------------------- 2022-08-17T13:35:31.7980332Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19496 2022-08-17T13:35:31.8009332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19497 2022-08-17T13:35:31.8038916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19498 2022-08-17T13:35:32.7850207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:32.7850606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:32.7853031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:32.7853426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:32.8258508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:32.8259046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:32.8262046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:32.8262493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:32.8319992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:32.8320355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:32.8324596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:32.8325200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:33.1971384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:33.1972008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:33.1972402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:33.1973067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:33.1973623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:33.1974161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:33.2079618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:33.2080287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:33.2287110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:35:33.2982635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:35:33.2983070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:33.2984796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:35:33.2985577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:33.2994166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:33.2994748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:33.2998162Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3098110Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3104284Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.3104785Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.3113093Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3113569Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3118545Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.3119026Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.3125316Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3126135Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:33.3129686Z STAGE:2022-08-17 13:35:33 19497:19497 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.3130154Z STAGE:2022-08-17 13:35:33 19498:19498 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:33.7086795Z ok (2.180s) 2022-08-17T13:35:33.7087077Z 2022-08-17T13:35:33.7087608Z ---------------------------------------------------------------------- 2022-08-17T13:35:33.7087945Z Ran 1 test in 2.180s 2022-08-17T13:35:33.7088068Z 2022-08-17T13:35:33.7088133Z OK 2022-08-17T13:35:33.7088217Z 2022-08-17T13:35:33.7089236Z Generating XML reports... 2022-08-17T13:35:33.7126305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133531.xml 2022-08-17T13:35:35.0156476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:35.0156910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:35.0159431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:35.0159827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:35.3552482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:35.3562186Z 2022-08-17T13:35:35.3562332Z Running tests... 2022-08-17T13:35:35.3562771Z ---------------------------------------------------------------------- 2022-08-17T13:35:35.6170708Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19557 2022-08-17T13:35:35.6199327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19558 2022-08-17T13:35:35.6227187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19559 2022-08-17T13:35:36.6342854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:36.6343581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:36.6346686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:36.6347296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:36.6395937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:36.6396392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:36.6400177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:36.6400694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:36.6630633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:36.6631318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:36.6634119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:36.6634724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:37.0151001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:37.0243242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:37.0243714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:37.0244355Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:37.0244929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:37.0252565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:37.0350774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:37.0351949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:37.0355636Z STAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.0356420Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1262830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:37.1266436Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1274768Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1275739Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1276132Z 2022-08-17T13:35:37.1284367Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1287184Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1287506Z 2022-08-17T13:35:37.1294074Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1294634Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1295295Z STAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1301181Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1303436Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:37.1303759Z 2022-08-17T13:35:37.1309402Z STAGE:2022-08-17 13:35:37 19557:19557 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1309903Z STAGE:2022-08-17 13:35:37 19558:19558 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.1310384Z STAGE:2022-08-17 13:35:37 19559:19559 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:37.5279450Z ok (2.171s) 2022-08-17T13:35:37.5279715Z 2022-08-17T13:35:37.5280266Z ---------------------------------------------------------------------- 2022-08-17T13:35:37.5280610Z Ran 1 test in 2.172s 2022-08-17T13:35:37.5280734Z 2022-08-17T13:35:37.5280785Z OK 2022-08-17T13:35:37.5280882Z 2022-08-17T13:35:37.5280977Z Generating XML reports... 2022-08-17T13:35:37.5318602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133535.xml 2022-08-17T13:35:38.8121320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:38.8121713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:38.8124379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:38.8124757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:39.1567528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:39.1577768Z 2022-08-17T13:35:39.1578030Z Running tests... 2022-08-17T13:35:39.1578390Z ---------------------------------------------------------------------- 2022-08-17T13:35:39.4252144Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19613 2022-08-17T13:35:39.4280557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19614 2022-08-17T13:35:39.4308693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19615 2022-08-17T13:35:40.4136536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:40.4137523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:40.4139936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:40.4140520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:40.4535719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:40.4536464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:40.4538101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:40.4538806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:40.4606693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:40.4607444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:40.4611594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:40.4612230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:40.8244622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:40.8345451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:40.8346126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:40.8347123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:40.8347724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:40.8348473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:40.8355266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:40.8355890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:40.8356502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:40.8362718Z STAGE:2022-08-17 13:35:40 19614:19614 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:40.8363477Z STAGE:2022-08-17 13:35:40 19613:19613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:40.8364876Z STAGE:2022-08-17 13:35:40 19615:19615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:40.8374543Z STAGE:2022-08-17 13:35:40 19614:19614 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:40.8375372Z STAGE:2022-08-17 13:35:40 19615:19615 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:40 19613:19613 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:40.8375682Z 2022-08-17T13:35:41.2356541Z ok (2.078s) 2022-08-17T13:35:41.2356706Z 2022-08-17T13:35:41.2357410Z ---------------------------------------------------------------------- 2022-08-17T13:35:41.2357835Z Ran 1 test in 2.078s 2022-08-17T13:35:41.2357963Z 2022-08-17T13:35:41.2358033Z OK 2022-08-17T13:35:41.2358128Z 2022-08-17T13:35:41.2358223Z Generating XML reports... 2022-08-17T13:35:41.2398059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133539.xml 2022-08-17T13:35:42.5517819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:42.5518211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:42.5520871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:42.5521235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:42.8954326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:42.8963735Z 2022-08-17T13:35:42.8963836Z Running tests... 2022-08-17T13:35:42.8964405Z ---------------------------------------------------------------------- 2022-08-17T13:35:43.1634569Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19669 2022-08-17T13:35:43.1662694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19670 2022-08-17T13:35:43.1690795Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19671 2022-08-17T13:35:44.1668562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:44.1669211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:44.1672046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:44.1672917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:44.1687905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:44.1688563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:44.1691570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:44.1692217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:44.1693056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:44.1693676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:44.1697258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:44.1697887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:44.5523869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:44.5624828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:44.5625584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:44.5626385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:44.5626960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:44.5627580Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:44.5633883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:44.5635378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:44.5635902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:44.5640228Z STAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5640959Z STAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5641419Z STAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5650058Z STAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5651049Z STAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5651363Z 2022-08-17T13:35:44.5670410Z STAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5677298Z STAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5677614Z 2022-08-17T13:35:44.5685083Z STAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5685748Z STAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5686237Z 2022-08-17T13:35:44.5698376Z STAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5704852Z STAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:44.5705290Z 2022-08-17T13:35:44.5710882Z STAGE:2022-08-17 13:35:44 19671:19671 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5712054Z STAGE:2022-08-17 13:35:44 19669:19669 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:44 19670:19670 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:44.5712567Z 2022-08-17T13:35:44.9738966Z ok (2.077s) 2022-08-17T13:35:44.9739208Z 2022-08-17T13:35:44.9739891Z ---------------------------------------------------------------------- 2022-08-17T13:35:44.9740209Z Ran 1 test in 2.077s 2022-08-17T13:35:44.9740345Z 2022-08-17T13:35:44.9740395Z OK 2022-08-17T13:35:44.9740491Z 2022-08-17T13:35:44.9740587Z Generating XML reports... 2022-08-17T13:35:44.9774941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133542.xml 2022-08-17T13:35:46.2674851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:46.2675239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:46.2678211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:46.2678621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:46.6146765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:46.6156230Z 2022-08-17T13:35:46.6156315Z Running tests... 2022-08-17T13:35:46.6156785Z ---------------------------------------------------------------------- 2022-08-17T13:35:46.6175033Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-08-17T13:35:46.6175485Z 2022-08-17T13:35:46.6175970Z ---------------------------------------------------------------------- 2022-08-17T13:35:46.6176301Z Ran 1 test in 0.002s 2022-08-17T13:35:46.6176419Z 2022-08-17T13:35:46.6176496Z OK (skipped=1) 2022-08-17T13:35:46.6176611Z 2022-08-17T13:35:46.6176700Z Generating XML reports... 2022-08-17T13:35:46.6201210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133546.xml 2022-08-17T13:35:47.7738612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:47.7739010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:47.7741227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:47.7741845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:48.1102747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:48.1112484Z 2022-08-17T13:35:48.1112585Z Running tests... 2022-08-17T13:35:48.1113140Z ---------------------------------------------------------------------- 2022-08-17T13:35:48.1131755Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-08-17T13:35:48.1132217Z 2022-08-17T13:35:48.1132641Z ---------------------------------------------------------------------- 2022-08-17T13:35:48.1133070Z Ran 1 test in 0.002s 2022-08-17T13:35:48.1133272Z 2022-08-17T13:35:48.1133401Z OK (skipped=1) 2022-08-17T13:35:48.1133588Z 2022-08-17T13:35:48.1133747Z Generating XML reports... 2022-08-17T13:35:48.1158920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133548.xml 2022-08-17T13:35:49.2631939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:49.2632334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:49.2634954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:49.2635323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:49.6067803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:49.6078407Z 2022-08-17T13:35:49.6078868Z Running tests... 2022-08-17T13:35:49.6079303Z ---------------------------------------------------------------------- 2022-08-17T13:35:49.8708910Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19745 2022-08-17T13:35:49.8735948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19746 2022-08-17T13:35:49.8764123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19747 2022-08-17T13:35:50.8404299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:50.8405015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:50.8406866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:50.8407538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:50.8410381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:50.8411050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:50.8415861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:50.8416492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:50.8432038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:50.8432442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:50.8436783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:50.8437161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:51.2081041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:51.2182450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:51.2183706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:51.2184391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:51.2184918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:51.2185477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:51.2190636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:51.2191335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:51.2192087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:51.2299427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:35:51.2401489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:35:51.2402300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:35:51.2403069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:51.2403890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:51.2404769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:51.2405616Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2406601Z STAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2407313Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2414864Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2415984Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2416497Z 2022-08-17T13:35:51.2432428Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2432885Z 2022-08-17T13:35:51.2433291Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2438320Z STAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2439001Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2439461Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2449343Z STAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2450127Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2450783Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:51.2455034Z STAGE:2022-08-17 13:35:51 19746:19746 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2455733Z STAGE:2022-08-17 13:35:51 19745:19745 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:51 19747:19747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:51.2456039Z 2022-08-17T13:35:51.6816825Z ok (2.074s) 2022-08-17T13:35:51.6817242Z 2022-08-17T13:35:51.6817816Z ---------------------------------------------------------------------- 2022-08-17T13:35:51.6818073Z Ran 1 test in 2.074s 2022-08-17T13:35:51.6818194Z 2022-08-17T13:35:51.6818261Z OK 2022-08-17T13:35:51.6818357Z 2022-08-17T13:35:51.6818452Z Generating XML reports... 2022-08-17T13:35:51.6852558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133549.xml 2022-08-17T13:35:52.9532635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:52.9533041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:52.9535631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:52.9536021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:53.2916763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:53.2926795Z 2022-08-17T13:35:53.2926931Z Running tests... 2022-08-17T13:35:53.2927619Z ---------------------------------------------------------------------- 2022-08-17T13:35:53.5585154Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19810 2022-08-17T13:35:53.5612630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19811 2022-08-17T13:35:53.5641520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19812 2022-08-17T13:35:54.5347781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:54.5348368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:54.5351496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:54.5352207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:54.5776814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:54.5777462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:54.5780710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:54.5781342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:54.5845571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:54.5849175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:54.5849920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:54.5850292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:54.9544679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:35:54.9545330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:35:54.9545729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:35:54.9546358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:54.9546918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:54.9547465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:35:54.9651995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:35:55.0557395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:35:55.0558319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:35:55.0558810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:35:55.0663352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:35:55.0664075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:35:55.0664827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:55.0665390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:55.0667976Z STAGE:2022-08-17 13:35:55 19811:19811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:55.0761857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:35:55.0768107Z STAGE:2022-08-17 13:35:55 19812:19812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:55.0775943Z STAGE:2022-08-17 13:35:55 19812:19812 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:55.0776714Z STAGE:2022-08-17 13:35:55 19811:19811 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:55.0792599Z STAGE:2022-08-17 13:35:55 19812:19812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:55.0793152Z STAGE:2022-08-17 13:35:55 19811:19811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:35:55.0797955Z STAGE:2022-08-17 13:35:55 19812:19812 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:35:55 19811:19811 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:35:55.0798303Z 2022-08-17T13:35:55.5692741Z ok (2.276s) 2022-08-17T13:35:55.5692971Z 2022-08-17T13:35:55.5693529Z ---------------------------------------------------------------------- 2022-08-17T13:35:55.5693877Z Ran 1 test in 2.277s 2022-08-17T13:35:55.5694003Z 2022-08-17T13:35:55.5694070Z OK 2022-08-17T13:35:55.5694168Z 2022-08-17T13:35:55.5694279Z Generating XML reports... 2022-08-17T13:35:55.5731749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133553.xml 2022-08-17T13:35:56.8766925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:56.8767592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:56.8770592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:56.8771238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:57.2179581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:57.2190192Z 2022-08-17T13:35:57.2190617Z Running tests... 2022-08-17T13:35:57.2191040Z ---------------------------------------------------------------------- 2022-08-17T13:35:57.2209738Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-08-17T13:35:57.2210192Z 2022-08-17T13:35:57.2210600Z ---------------------------------------------------------------------- 2022-08-17T13:35:57.2211003Z Ran 1 test in 0.002s 2022-08-17T13:35:57.2211201Z 2022-08-17T13:35:57.2211332Z OK (skipped=1) 2022-08-17T13:35:57.2211535Z 2022-08-17T13:35:57.2211699Z Generating XML reports... 2022-08-17T13:35:57.2236522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133557.xml 2022-08-17T13:35:58.3765599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:58.3766565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:58.3768852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:58.3769357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:35:58.7236709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:35:58.7247093Z 2022-08-17T13:35:58.7247228Z Running tests... 2022-08-17T13:35:58.7247892Z ---------------------------------------------------------------------- 2022-08-17T13:35:58.7266993Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-08-17T13:35:58.7267410Z 2022-08-17T13:35:58.7267831Z ---------------------------------------------------------------------- 2022-08-17T13:35:58.7268273Z Ran 1 test in 0.002s 2022-08-17T13:35:58.7268394Z 2022-08-17T13:35:58.7268472Z OK (skipped=1) 2022-08-17T13:35:58.7268589Z 2022-08-17T13:35:58.7268681Z Generating XML reports... 2022-08-17T13:35:58.7294108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133558.xml 2022-08-17T13:35:59.8978862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:35:59.8979286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:35:59.8981178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:35:59.8981559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:00.2401754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:00.2410909Z 2022-08-17T13:36:00.2411060Z Running tests... 2022-08-17T13:36:00.2411478Z ---------------------------------------------------------------------- 2022-08-17T13:36:00.5059053Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19891 2022-08-17T13:36:00.5086691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19892 2022-08-17T13:36:00.5114910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19893 2022-08-17T13:36:01.5233330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:01.5234023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:01.5236185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:01.5236834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:01.5324549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:01.5325163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:01.5329559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:01.5330254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:01.5900134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:01.5900793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:01.5904838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:01.5905462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:01.9670938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:01.9772172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:01.9772965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:01.9773432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:01.9774285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:01.9775162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:01.9781438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:01.9782012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:01.9782647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:02.3163013Z ok (2.075s) 2022-08-17T13:36:02.3163162Z 2022-08-17T13:36:02.3163542Z ---------------------------------------------------------------------- 2022-08-17T13:36:02.3163822Z Ran 1 test in 2.075s 2022-08-17T13:36:02.3164198Z 2022-08-17T13:36:02.3164267Z OK 2022-08-17T13:36:02.3164370Z 2022-08-17T13:36:02.3164455Z Generating XML reports... 2022-08-17T13:36:02.3198679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133600.xml 2022-08-17T13:36:03.6159451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:03.6159848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:03.6162187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:03.6162571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:03.9624324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:03.9633994Z 2022-08-17T13:36:03.9634100Z Running tests... 2022-08-17T13:36:03.9634580Z ---------------------------------------------------------------------- 2022-08-17T13:36:04.2291005Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19944 2022-08-17T13:36:04.2318529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19945 2022-08-17T13:36:04.2347301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19946 2022-08-17T13:36:05.2219937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:05.2220575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:05.2222510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:05.2223158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:05.2298984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:05.2299692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:05.2300524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:05.2301060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:05.2302633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:05.2303249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:05.2304042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:05.2304914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:05.6064055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:05.6064595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:05.6065216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:05.6066185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:05.6066782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:05.6067337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:05.6072588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:05.6073699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:05.6074729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:05.6590132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:05.6590545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:05.6691165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:05.6692161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:05.6693000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:05.6693856Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:05.6842593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-08-17T13:36:05.6942095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-08-17T13:36:05.6942672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-08-17T13:36:05.6943336Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:36:05.6943956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:36:05.6944595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes. 2022-08-17T13:36:05.7266897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-08-17T13:36:05.7368854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-08-17T13:36:05.7369513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-08-17T13:36:05.7370454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:36:05.7371320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:36:05.7372140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes. 2022-08-17T13:36:06.1396856Z ok (2.176s) 2022-08-17T13:36:06.1397074Z 2022-08-17T13:36:06.1397943Z ---------------------------------------------------------------------- 2022-08-17T13:36:06.1398522Z Ran 1 test in 2.176s 2022-08-17T13:36:06.1398647Z 2022-08-17T13:36:06.1398705Z OK 2022-08-17T13:36:06.1398803Z 2022-08-17T13:36:06.1398900Z Generating XML reports... 2022-08-17T13:36:06.1436524Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133603.xml 2022-08-17T13:36:07.4438257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:07.4438647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:07.4441034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:07.4441501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:07.7839817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:07.7849380Z 2022-08-17T13:36:07.7849889Z Running tests... 2022-08-17T13:36:07.7850333Z ---------------------------------------------------------------------- 2022-08-17T13:36:08.0496105Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20033 2022-08-17T13:36:08.0524682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20034 2022-08-17T13:36:08.0553463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20035 2022-08-17T13:36:09.0333556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:09.0333961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:09.0337466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:09.0338129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:09.0424602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:09.0425276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:09.0428354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:09.0429005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:09.0560282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:09.0560916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:09.0563864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:09.0564527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:09.4153481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:09.4154089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:09.4154479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:09.4155131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:09.4155691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:09.4156227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:09.4259697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:09.5166178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:09.5166992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:09.5372152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:09.5473999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:09.5474425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:09.5475074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:09.5475926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:09.5476533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:09.5478289Z STAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5478734Z 2022-08-17T13:36:09.5578947Z STAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5586457Z STAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5587326Z STAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5587644Z 2022-08-17T13:36:09.5601541Z STAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5608782Z STAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5609492Z STAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5616412Z STAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5617529Z STAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5618304Z 2022-08-17T13:36:09.5626891Z STAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5631165Z STAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5631937Z STAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:09.5652752Z STAGE:2022-08-17 13:36:09 20034:20034 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:09 20033:20033 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5653801Z STAGE:2022-08-17 13:36:09 20035:20035 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:09.5654149Z 2022-08-17T13:36:09.9604451Z ok (2.175s) 2022-08-17T13:36:09.9604707Z 2022-08-17T13:36:09.9605224Z ---------------------------------------------------------------------- 2022-08-17T13:36:09.9605491Z Ran 1 test in 2.175s 2022-08-17T13:36:09.9605625Z 2022-08-17T13:36:09.9605692Z OK 2022-08-17T13:36:09.9605790Z 2022-08-17T13:36:09.9606029Z Generating XML reports... 2022-08-17T13:36:09.9642933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133607.xml 2022-08-17T13:36:11.2539436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:11.2539843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:11.2542458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:11.2542839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:11.5953326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:11.5962420Z 2022-08-17T13:36:11.5962559Z Running tests... 2022-08-17T13:36:11.5963031Z ---------------------------------------------------------------------- 2022-08-17T13:36:11.8575129Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20098 2022-08-17T13:36:11.8604108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20099 2022-08-17T13:36:11.8632165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20100 2022-08-17T13:36:12.8353341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:12.8353736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:12.8356654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:12.8357626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:12.8358471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:12.8359068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:12.8362005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:12.8362474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:12.8801039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:12.8801430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:12.8803721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:12.8804252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:13.2428728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:13.2528766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:13.2529370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:13.2530042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:13.2530604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:13.2531165Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:13.2537650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:13.2538560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:13.2539196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:13.2747100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:13.2747975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:13.2748686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:13.2749693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:13.2750857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:13.2751703Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:13.2752355Z STAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2752825Z STAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2851148Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2859072Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2859598Z 2022-08-17T13:36:13.2859609Z 2022-08-17T13:36:13.2875200Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2876224Z STAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2876667Z 2022-08-17T13:36:13.2883544Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2884022Z STAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2884488Z STAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2894170Z STAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2894974Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2895605Z STAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:13.2900701Z STAGE:2022-08-17 13:36:13 20100:20100 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2901278Z STAGE:2022-08-17 13:36:13 20098:20098 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.2901821Z STAGE:2022-08-17 13:36:13 20099:20099 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:13.6680793Z ok (2.071s) 2022-08-17T13:36:13.6681110Z 2022-08-17T13:36:13.6681660Z ---------------------------------------------------------------------- 2022-08-17T13:36:13.6690268Z Ran 1 test in 2.072s 2022-08-17T13:36:13.6690465Z 2022-08-17T13:36:13.6690535Z OK 2022-08-17T13:36:13.6690623Z 2022-08-17T13:36:13.6690726Z Generating XML reports... 2022-08-17T13:36:13.6717668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133611.xml 2022-08-17T13:36:14.9769472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:14.9769892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:14.9772609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:14.9773175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:15.3213174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:15.3223149Z 2022-08-17T13:36:15.3223297Z Running tests... 2022-08-17T13:36:15.3223933Z ---------------------------------------------------------------------- 2022-08-17T13:36:15.5900094Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20163 2022-08-17T13:36:15.5930389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20164 2022-08-17T13:36:15.5960931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20165 2022-08-17T13:36:16.5725447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:16.5726383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:16.5728066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:16.5728701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:16.6125696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:16.6126321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:16.6128916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:16.6129297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:16.6225804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:16.6226182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:16.6229538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:16.6229962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:16.9920786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:16.9921436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:16.9922041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:16.9923002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:16.9942426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:16.9943026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:17.0027690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:17.0934987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:17.0935567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:17.1142034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:17.1244587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:17.1245585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:17.1246202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:17.1246749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:17.1247495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:17.1248579Z STAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1249381Z STAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1249856Z 2022-08-17T13:36:17.1257383Z STAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1258055Z STAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1258364Z 2022-08-17T13:36:17.1273182Z STAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1279536Z STAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1279853Z 2022-08-17T13:36:17.1287107Z STAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1287787Z STAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1288101Z 2022-08-17T13:36:17.1297658Z STAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1301457Z STAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:17.1301899Z 2022-08-17T13:36:17.1308862Z STAGE:2022-08-17 13:36:17 20165:20165 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1310028Z STAGE:2022-08-17 13:36:17 20164:20164 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:17 20163:20163 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:17.1310564Z 2022-08-17T13:36:17.5014649Z ok (2.179s) 2022-08-17T13:36:17.5014910Z 2022-08-17T13:36:17.5015436Z ---------------------------------------------------------------------- 2022-08-17T13:36:17.5015705Z Ran 1 test in 2.179s 2022-08-17T13:36:17.5015831Z 2022-08-17T13:36:17.5015897Z OK 2022-08-17T13:36:17.5015995Z 2022-08-17T13:36:17.5016093Z Generating XML reports... 2022-08-17T13:36:17.5052461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133615.xml 2022-08-17T13:36:18.8366818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:18.8367232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:18.8369664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:18.8370148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:19.1815872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:19.1825723Z 2022-08-17T13:36:19.1825834Z Running tests... 2022-08-17T13:36:19.1826455Z ---------------------------------------------------------------------- 2022-08-17T13:36:19.4514487Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20228 2022-08-17T13:36:19.4542214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20229 2022-08-17T13:36:19.4569967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20230 2022-08-17T13:36:20.4435396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:20.4436083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:20.4439034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:20.4439950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:20.4581826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:20.4582590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:20.4585624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:20.4586295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:20.4743923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:20.4744563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:20.4747387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:20.4748056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:20.8374878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:20.8476187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:20.8476711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:20.8477129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:20.8477678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:20.8478586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:20.8484099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:20.8484840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:20.9487094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:20.9597271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:20.9699243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:20.9699856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:20.9700785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:20.9701706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:20.9702682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:20.9704167Z STAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9704835Z STAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9705143Z 2022-08-17T13:36:20.9715067Z STAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9715717Z STAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9715934Z 2022-08-17T13:36:20.9733893Z STAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9734741Z 2022-08-17T13:36:20.9734749Z 2022-08-17T13:36:20.9741695Z STAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9742844Z STAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9743364Z 2022-08-17T13:36:20.9753794Z STAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9754287Z STAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9754728Z STAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:20.9761440Z STAGE:2022-08-17 13:36:20 20230:20230 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9762799Z STAGE:2022-08-17 13:36:20 20229:20229 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:20 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:20.9763335Z 2022-08-17T13:36:21.3619884Z ok (2.179s) 2022-08-17T13:36:21.3620083Z 2022-08-17T13:36:21.3620567Z ---------------------------------------------------------------------- 2022-08-17T13:36:21.3620833Z Ran 1 test in 2.179s 2022-08-17T13:36:21.3620956Z 2022-08-17T13:36:21.3621006Z OK 2022-08-17T13:36:21.3621105Z 2022-08-17T13:36:21.3621272Z Generating XML reports... 2022-08-17T13:36:21.3655745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133619.xml 2022-08-17T13:36:22.6287792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:22.6288197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:22.6290393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:22.6290764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:22.9693043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:22.9703808Z 2022-08-17T13:36:22.9704066Z Running tests... 2022-08-17T13:36:22.9704695Z ---------------------------------------------------------------------- 2022-08-17T13:36:23.2317257Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20293 2022-08-17T13:36:23.2344519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20294 2022-08-17T13:36:23.2371990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20295 2022-08-17T13:36:24.2203343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:24.2204042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:24.2206500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:24.2207135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:24.2564954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:24.2565645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:24.2569173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:24.2569835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:24.2570970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:24.2571525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:24.2574368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:24.2575011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:24.6242559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:24.6243047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:24.6243702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:24.6244140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:24.6244672Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:24.6245219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:24.6349785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:24.6350204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:24.6555458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:24.6555891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:24.7253291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:24.7255780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:24.7256465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:24.7263889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:24.7264473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:24.7267707Z STAGE:2022-08-17 13:36:24 20294:20294 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:24.7367588Z STAGE:2022-08-17 13:36:24 20295:20295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:24.7374944Z STAGE:2022-08-17 13:36:24 20295:20295 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:24.7375538Z STAGE:2022-08-17 13:36:24 20294:20294 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:24.7390477Z STAGE:2022-08-17 13:36:24 20295:20295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:24.7390940Z STAGE:2022-08-17 13:36:24 20294:20294 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:24.7396597Z STAGE:2022-08-17 13:36:24 20295:20295 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:24 20294:20294 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:24.7396910Z 2022-08-17T13:36:25.1423218Z ok (2.172s) 2022-08-17T13:36:25.1423451Z 2022-08-17T13:36:25.1423923Z ---------------------------------------------------------------------- 2022-08-17T13:36:25.1424352Z Ran 1 test in 2.172s 2022-08-17T13:36:25.1424540Z 2022-08-17T13:36:25.1424636Z OK 2022-08-17T13:36:25.1424791Z 2022-08-17T13:36:25.1424939Z Generating XML reports... 2022-08-17T13:36:25.1460024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133622.xml 2022-08-17T13:36:26.4329853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:26.4330245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:26.4332949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:26.4333332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:26.7782490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:26.7792647Z 2022-08-17T13:36:26.7793098Z Running tests... 2022-08-17T13:36:26.7793529Z ---------------------------------------------------------------------- 2022-08-17T13:36:27.0498349Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20354 2022-08-17T13:36:27.0526868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20355 2022-08-17T13:36:27.0555978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20356 2022-08-17T13:36:28.0746393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:28.0747123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:28.0749775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:28.0750473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:28.0843361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:28.0843987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:28.0848370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:28.0848999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:28.0849815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:28.0850415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:28.0854304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:28.0854947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:28.4476067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:28.4577119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:28.4577653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:28.4578305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:28.4578873Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:28.4580037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:28.4586212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:28.4588437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:28.4589051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:28.4589605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:28.4694738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:28.4695857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:28.4696542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:28.4700016Z STAGE:2022-08-17 13:36:28 20355:20355 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:28.4792275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:28.4796793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:28.4800535Z STAGE:2022-08-17 13:36:28 20356:20356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:28.4808504Z STAGE:2022-08-17 13:36:28 20356:20356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:28.4809231Z STAGE:2022-08-17 13:36:28 20355:20355 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:28.4824175Z STAGE:2022-08-17 13:36:28 20356:20356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:28.4824856Z STAGE:2022-08-17 13:36:28 20355:20355 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:28.4830173Z STAGE:2022-08-17 13:36:28 20356:20356 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:28.4830662Z STAGE:2022-08-17 13:36:28 20355:20355 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:28.9608057Z ok (2.181s) 2022-08-17T13:36:28.9608274Z 2022-08-17T13:36:28.9608610Z ---------------------------------------------------------------------- 2022-08-17T13:36:28.9609033Z Ran 1 test in 2.181s 2022-08-17T13:36:28.9609146Z 2022-08-17T13:36:28.9609214Z OK 2022-08-17T13:36:28.9609312Z 2022-08-17T13:36:28.9609411Z Generating XML reports... 2022-08-17T13:36:28.9644469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133626.xml 2022-08-17T13:36:30.2683566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:30.2683984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:30.2686503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:30.2686867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:30.6194549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:30.6204325Z 2022-08-17T13:36:30.6204493Z Running tests... 2022-08-17T13:36:30.6204967Z ---------------------------------------------------------------------- 2022-08-17T13:36:30.8921758Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20415 2022-08-17T13:36:30.8952375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20416 2022-08-17T13:36:30.8982639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20417 2022-08-17T13:36:31.8825820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:31.8826404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:31.8829325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:31.8829973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:31.8876047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:31.8876725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:31.8879723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:31.8880129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:31.9320947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:31.9321340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:31.9324953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:31.9325549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:32.3042970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:32.3144981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:32.3145829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:32.3146490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:32.3147210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:32.3147750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:32.3153784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:32.3154605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:32.3155300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:32.3156271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:32.3361450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:32.3361998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:32.3362648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:32.3363220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:32.3365781Z STAGE:2022-08-17 13:36:32 20416:20416 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:32.3461081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:32.3466488Z STAGE:2022-08-17 13:36:32 20417:20417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:32.3473619Z STAGE:2022-08-17 13:36:32 20417:20417 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:32.3474348Z STAGE:2022-08-17 13:36:32 20416:20416 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:32.3489446Z STAGE:2022-08-17 13:36:32 20417:20417 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:32.3490694Z STAGE:2022-08-17 13:36:32 20416:20416 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:32.3495984Z STAGE:2022-08-17 13:36:32 20416:20416 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:32.3496529Z STAGE:2022-08-17 13:36:32 20417:20417 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:32.8036410Z ok (2.183s) 2022-08-17T13:36:32.8036665Z 2022-08-17T13:36:32.8037056Z ---------------------------------------------------------------------- 2022-08-17T13:36:32.8037322Z Ran 1 test in 2.183s 2022-08-17T13:36:32.8037441Z 2022-08-17T13:36:32.8037718Z OK 2022-08-17T13:36:32.8037820Z 2022-08-17T13:36:32.8037919Z Generating XML reports... 2022-08-17T13:36:32.8072825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133630.xml 2022-08-17T13:36:34.0955322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:34.0955714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:34.0958183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:34.0958568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:34.4383488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:34.4393650Z 2022-08-17T13:36:34.4393780Z Running tests... 2022-08-17T13:36:34.4394422Z ---------------------------------------------------------------------- 2022-08-17T13:36:34.7042544Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20476 2022-08-17T13:36:34.7072482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20477 2022-08-17T13:36:34.7102120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20478 2022-08-17T13:36:35.7648867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:35.7649489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:35.7652537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:35.7781651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:35.7782722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:35.7783326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:35.7802164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:35.7802580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:35.7873294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:35.7873964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:35.7877802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:35.7878404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:36.1580703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:36.1681359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:36.1681987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:36.1682866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:36.1683799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:36.1684419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:36.1690135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:36.1690870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:36.1691875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:36.1692663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:36:36.1902441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:36:36.1903157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:36:36.1903800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:36.1904359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:36.1906546Z STAGE:2022-08-17 13:36:36 20477:20477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:36.1997703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:36:36.2007331Z STAGE:2022-08-17 13:36:36 20478:20478 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:36.2015172Z STAGE:2022-08-17 13:36:36 20478:20478 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:36.2015928Z STAGE:2022-08-17 13:36:36 20477:20477 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:36.2033451Z STAGE:2022-08-17 13:36:36 20478:20478 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:36.2033995Z STAGE:2022-08-17 13:36:36 20477:20477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:36.2039969Z STAGE:2022-08-17 13:36:36 20477:20477 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:36 20478:20478 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:36.2040290Z 2022-08-17T13:36:36.6150787Z ok (2.175s) 2022-08-17T13:36:36.6151077Z 2022-08-17T13:36:36.6151548Z ---------------------------------------------------------------------- 2022-08-17T13:36:36.6151921Z Ran 1 test in 2.176s 2022-08-17T13:36:36.6152122Z 2022-08-17T13:36:36.6152267Z OK 2022-08-17T13:36:36.6152441Z 2022-08-17T13:36:36.6152551Z Generating XML reports... 2022-08-17T13:36:36.6187232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133634.xml 2022-08-17T13:36:37.9092363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:37.9092745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:37.9095720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:37.9096129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:38.2477852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:38.2488256Z 2022-08-17T13:36:38.2488571Z Running tests... 2022-08-17T13:36:38.2489207Z ---------------------------------------------------------------------- 2022-08-17T13:36:38.5107251Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20537 2022-08-17T13:36:38.5135328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20538 2022-08-17T13:36:38.5164835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20539 2022-08-17T13:36:39.5019666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:39.5020309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:39.5029424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:39.5030088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:39.5112537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:39.5112935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:39.5115633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:39.5116007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:39.5121326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:39.5121690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:39.5126419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:39.5126789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:39.8948521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:39.9049949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:39.9050681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:39.9051660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:39.9052235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:39.9052772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:39.9060624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:39.9065200Z STAGE:2022-08-17 13:36:39 20538:20538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0064217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:40.0064858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:40.0068305Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0068793Z STAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0076590Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0077057Z 2022-08-17T13:36:40.0077499Z STAGE:2022-08-17 13:36:40 20538:20538 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0093440Z STAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0094051Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0094555Z STAGE:2022-08-17 13:36:40 20538:20538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0101833Z STAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0103049Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:40 20538:20538 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0103554Z 2022-08-17T13:36:40.0110700Z STAGE:2022-08-17 13:36:40 20538:20538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0111394Z STAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0112408Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:40.0119025Z STAGE:2022-08-17 13:36:40 20539:20539 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0119860Z STAGE:2022-08-17 13:36:40 20537:20537 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.0120716Z STAGE:2022-08-17 13:36:40 20538:20538 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:40.4213909Z ok (2.172s) 2022-08-17T13:36:40.4214158Z 2022-08-17T13:36:40.4214633Z ---------------------------------------------------------------------- 2022-08-17T13:36:40.4214922Z Ran 1 test in 2.173s 2022-08-17T13:36:40.4215048Z 2022-08-17T13:36:40.4215114Z OK 2022-08-17T13:36:40.4215214Z 2022-08-17T13:36:40.4215295Z Generating XML reports... 2022-08-17T13:36:40.4250295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133638.xml 2022-08-17T13:36:41.7100005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:41.7100422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:41.7103098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:41.7103715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:42.0508745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:42.0518432Z 2022-08-17T13:36:42.0518903Z Running tests... 2022-08-17T13:36:42.0519324Z ---------------------------------------------------------------------- 2022-08-17T13:36:42.3112737Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20593 2022-08-17T13:36:42.3140973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20594 2022-08-17T13:36:42.3169785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20595 2022-08-17T13:36:43.2856952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:43.2857626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:43.2860581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:43.2860992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:43.3009872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:43.3010255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:43.3013342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:43.3013889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:43.3138804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:43.3139275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:43.3142084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:43.3142697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:43.6786299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:43.6888447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:43.6889142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:43.6890140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:43.6890923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:43.6891578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:43.6898078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:43.6898838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:43.6899869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:44.0220066Z ok (1.970s) 2022-08-17T13:36:44.0220309Z 2022-08-17T13:36:44.0220783Z ---------------------------------------------------------------------- 2022-08-17T13:36:44.0221232Z Ran 1 test in 1.970s 2022-08-17T13:36:44.0221434Z 2022-08-17T13:36:44.0221525Z OK 2022-08-17T13:36:44.0221712Z 2022-08-17T13:36:44.0221871Z Generating XML reports... 2022-08-17T13:36:44.0256143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133642.xml 2022-08-17T13:36:45.3334190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:45.3334613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:45.3336766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:45.3337147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:45.6739247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:45.6748899Z 2022-08-17T13:36:45.6749326Z Running tests... 2022-08-17T13:36:45.6749746Z ---------------------------------------------------------------------- 2022-08-17T13:36:45.9392529Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20646 2022-08-17T13:36:45.9421693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20647 2022-08-17T13:36:45.9450928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20648 2022-08-17T13:36:46.9202100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:46.9202738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:46.9204345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:46.9204990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:46.9309969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:46.9310709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:46.9314213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:46.9314851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:46.9437227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:46.9437882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:46.9440305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:46.9440906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:47.2985615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:47.3087601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:47.3088302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:47.3089263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:47.3090189Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:47.3090766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:47.3096389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:47.3097011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:47.3098232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:47.3101500Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3102482Z STAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3103464Z STAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3110961Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3111638Z STAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3111954Z 2022-08-17T13:36:47.3125576Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3132621Z STAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3133069Z 2022-08-17T13:36:47.3140606Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3141349Z STAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3141656Z 2022-08-17T13:36:47.3150272Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3155094Z STAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:47.3155609Z 2022-08-17T13:36:47.3162248Z STAGE:2022-08-17 13:36:47 20648:20648 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3162965Z STAGE:2022-08-17 13:36:47 20647:20647 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.3163432Z STAGE:2022-08-17 13:36:47 20646:20646 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:47.7499028Z ok (2.075s) 2022-08-17T13:36:47.7499193Z 2022-08-17T13:36:47.7499606Z ---------------------------------------------------------------------- 2022-08-17T13:36:47.7499886Z Ran 1 test in 2.075s 2022-08-17T13:36:47.7500008Z 2022-08-17T13:36:47.7500059Z OK 2022-08-17T13:36:47.7500156Z 2022-08-17T13:36:47.7500316Z Generating XML reports... 2022-08-17T13:36:47.7535972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133645.xml 2022-08-17T13:36:49.0691864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:49.0692309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:49.0695175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:49.0695587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:49.4112719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:49.4122517Z 2022-08-17T13:36:49.4123014Z Running tests... 2022-08-17T13:36:49.4123424Z ---------------------------------------------------------------------- 2022-08-17T13:36:49.6760757Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20702 2022-08-17T13:36:49.6789890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20703 2022-08-17T13:36:49.6818165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20704 2022-08-17T13:36:50.6420667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:50.6421322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:50.6423200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:50.6424105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:50.6448937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:50.6449584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:50.6453274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:50.6453685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:50.6862154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:50.6862665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:50.6865908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:50.6866286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:51.0471980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:51.0572345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:51.0572967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:51.0574163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:51.0574764Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:51.0575319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:51.0681174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:51.0681898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:51.0684639Z STAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.0688377Z STAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1585022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:51.1588650Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1596180Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1597522Z STAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1597975Z 2022-08-17T13:36:51.1612210Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1614140Z STAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1614487Z 2022-08-17T13:36:51.1621540Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1622073Z 2022-08-17T13:36:51.1622648Z STAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1630661Z STAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1631527Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1632331Z STAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:51.1638739Z STAGE:2022-08-17 13:36:51 20703:20703 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1639193Z STAGE:2022-08-17 13:36:51 20704:20704 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.1639662Z STAGE:2022-08-17 13:36:51 20702:20702 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:51.4870170Z ok (2.074s) 2022-08-17T13:36:51.4870439Z 2022-08-17T13:36:51.4870975Z ---------------------------------------------------------------------- 2022-08-17T13:36:51.4871373Z Ran 1 test in 2.075s 2022-08-17T13:36:51.4871509Z 2022-08-17T13:36:51.4871573Z OK 2022-08-17T13:36:51.4871670Z 2022-08-17T13:36:51.4871764Z Generating XML reports... 2022-08-17T13:36:51.4906875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133649.xml 2022-08-17T13:36:52.7816154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:52.7816828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:52.7819371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:52.7819970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:53.1259575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:53.1269177Z 2022-08-17T13:36:53.1269321Z Running tests... 2022-08-17T13:36:53.1270214Z ---------------------------------------------------------------------- 2022-08-17T13:36:53.3950142Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20758 2022-08-17T13:36:53.3980096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20759 2022-08-17T13:36:53.4008742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20760 2022-08-17T13:36:54.3909450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:54.3910100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:54.3912186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:54.3912803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:54.4307226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:54.4308146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:54.4310749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:54.4311352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:54.4481521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:54.4482201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:54.4484770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:54.4485412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:54.8175953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:54.8176652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:54.8177218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:54.8178068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:54.8178854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:54.8179828Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:54.8185261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:54.8185847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:54.8190539Z STAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.8190870Z 2022-08-17T13:36:54.9187067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:54.9191837Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9201826Z STAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9202570Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9203124Z STAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9220799Z STAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9228580Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9228903Z 2022-08-17T13:36:54.9237102Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9238166Z STAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9238679Z 2022-08-17T13:36:54.9250052Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9250634Z STAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9251276Z STAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:36:54.9257862Z STAGE:2022-08-17 13:36:54 20760:20760 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:36:54 20759:20759 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9259180Z STAGE:2022-08-17 13:36:54 20758:20758 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:36:54.9259559Z 2022-08-17T13:36:55.3059352Z ok (2.179s) 2022-08-17T13:36:55.3059511Z 2022-08-17T13:36:55.3060014Z ---------------------------------------------------------------------- 2022-08-17T13:36:55.3060501Z Ran 1 test in 2.179s 2022-08-17T13:36:55.3060711Z 2022-08-17T13:36:55.3060813Z OK 2022-08-17T13:36:55.3060961Z 2022-08-17T13:36:55.3061060Z Generating XML reports... 2022-08-17T13:36:55.3094990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133653.xml 2022-08-17T13:36:56.6046559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:56.6046977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:56.6049721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:56.6050106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:56.9446678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:36:56.9456748Z 2022-08-17T13:36:56.9456853Z Running tests... 2022-08-17T13:36:56.9457439Z ---------------------------------------------------------------------- 2022-08-17T13:36:57.2077420Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20814 2022-08-17T13:36:57.2105685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20815 2022-08-17T13:36:57.2133096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20816 2022-08-17T13:36:58.1818138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:58.1818892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:58.1821591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:58.1822224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:58.2039995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:58.2040693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:58.2044244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:58.2044760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:58.2263354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:36:58.2264006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:36:58.2266618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:36:58.2267278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:36:58.5889868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:36:58.5990555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:36:58.5991294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:36:58.5992401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:58.5993317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:58.5993868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:36:58.6097579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:36:58.7002610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:36:58.7003031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:36:59.0181470Z ok (2.072s) 2022-08-17T13:36:59.0181687Z 2022-08-17T13:36:59.0182134Z ---------------------------------------------------------------------- 2022-08-17T13:36:59.0182643Z Ran 1 test in 2.072s 2022-08-17T13:36:59.0182844Z 2022-08-17T13:36:59.0182937Z OK 2022-08-17T13:36:59.0183084Z 2022-08-17T13:36:59.0183235Z Generating XML reports... 2022-08-17T13:36:59.0218760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133656.xml 2022-08-17T13:37:00.3254879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:00.3255860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:00.3258115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:00.3258715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:00.6722352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:00.6732898Z 2022-08-17T13:37:00.6733388Z Running tests... 2022-08-17T13:37:00.6733811Z ---------------------------------------------------------------------- 2022-08-17T13:37:00.9427199Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20867 2022-08-17T13:37:00.9456902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20868 2022-08-17T13:37:00.9485686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20869 2022-08-17T13:37:01.9330244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:01.9330918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:01.9333135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:01.9333767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:01.9617522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:01.9618146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:01.9620683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:01.9621077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:01.9860118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:01.9860510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:01.9863785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:01.9864348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:02.3522729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:02.3624049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:02.3624833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:02.3625667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:02.3626229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:02.3626780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:02.3632632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:02.3633911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:02.3634422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:02.3839211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:02.3941254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:02.3941986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:02.3943409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:02.3944058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:02.3944828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:02.3945709Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3946156Z STAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3952552Z STAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3958058Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3958617Z STAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3959089Z STAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3967712Z STAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3968509Z STAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3969255Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3976280Z STAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3977325Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3977810Z 2022-08-17T13:37:02.3983352Z STAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3984437Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3985177Z STAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:02.3991401Z STAGE:2022-08-17 13:37:02 20867:20867 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3992490Z STAGE:2022-08-17 13:37:02 20869:20869 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:02 20868:20868 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:02.3993053Z 2022-08-17T13:37:02.8535437Z ok (2.180s) 2022-08-17T13:37:02.8535666Z 2022-08-17T13:37:02.8536133Z ---------------------------------------------------------------------- 2022-08-17T13:37:02.8536553Z Ran 1 test in 2.180s 2022-08-17T13:37:02.8536752Z 2022-08-17T13:37:02.8536833Z OK 2022-08-17T13:37:02.8536976Z 2022-08-17T13:37:02.8537146Z Generating XML reports... 2022-08-17T13:37:02.8572624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133700.xml 2022-08-17T13:37:04.1483013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:04.1483445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:04.1486312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:04.1486751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:04.4911361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:04.4921037Z 2022-08-17T13:37:04.4921142Z Running tests... 2022-08-17T13:37:04.4922048Z ---------------------------------------------------------------------- 2022-08-17T13:37:04.7623856Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20932 2022-08-17T13:37:04.7653656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20933 2022-08-17T13:37:04.7684697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20934 2022-08-17T13:37:05.7270471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:05.7271121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:05.7272412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:05.7272828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:05.7744914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:05.7745563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:05.7748252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:05.7748813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:05.7757712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:05.7758361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:05.7762355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:05.7763187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:06.1451653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:06.1452233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:06.1452855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:06.1453865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:06.1454477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:06.1455026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:06.1461603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:06.1462462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:06.2464505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:06.2771890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:06.2874200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:06.2874785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:06.2875733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:06.2876684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:06.2877496Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:06.2878459Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2879195Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2880140Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2886957Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2887727Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2888463Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2897401Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2898100Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2898697Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2904335Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2904820Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2905282Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2910789Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2912925Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2913383Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:06.2919869Z STAGE:2022-08-17 13:37:06 20934:20934 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2920400Z STAGE:2022-08-17 13:37:06 20932:20932 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.2920936Z STAGE:2022-08-17 13:37:06 20933:20933 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:06.6737464Z ok (2.181s) 2022-08-17T13:37:06.6737686Z 2022-08-17T13:37:06.6738007Z ---------------------------------------------------------------------- 2022-08-17T13:37:06.6738275Z Ran 1 test in 2.182s 2022-08-17T13:37:06.6738443Z 2022-08-17T13:37:06.6738508Z OK 2022-08-17T13:37:06.6738603Z 2022-08-17T13:37:06.6738700Z Generating XML reports... 2022-08-17T13:37:06.6774175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133704.xml 2022-08-17T13:37:07.9997427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:07.9998061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:07.9999949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:08.0000455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:08.3468329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:08.3478364Z 2022-08-17T13:37:08.3478624Z Running tests... 2022-08-17T13:37:08.3479236Z ---------------------------------------------------------------------- 2022-08-17T13:37:08.6187306Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20997 2022-08-17T13:37:08.6218033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20998 2022-08-17T13:37:08.6248084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20999 2022-08-17T13:37:09.6282627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:09.6283032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:09.6286076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:09.6286543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:09.6330276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:09.6330636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:09.6334780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:09.6335146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:09.6592238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:09.6592639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:09.6596167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:09.6596528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:10.0282159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:10.0383061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:10.0383799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:10.0384450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:10.0385028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:10.0385568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:10.0491603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:10.1395994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:10.1396678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:10.1604719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:10.1706262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:10.1706881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:10.1708118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:10.1709021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:10.1709858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:10.1710525Z STAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1720817Z STAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1811204Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1817485Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1818021Z STAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1818477Z STAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1827836Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1830872Z STAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1831182Z 2022-08-17T13:37:10.1837587Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1838247Z STAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1838551Z 2022-08-17T13:37:10.1844066Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1846832Z STAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:10.1847146Z 2022-08-17T13:37:10.1853564Z STAGE:2022-08-17 13:37:10 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1854270Z STAGE:2022-08-17 13:37:10 20999:20999 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:10 20998:20998 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:10.1854580Z 2022-08-17T13:37:10.6302417Z ok (2.282s) 2022-08-17T13:37:10.6302692Z 2022-08-17T13:37:10.6303139Z ---------------------------------------------------------------------- 2022-08-17T13:37:10.6303432Z Ran 1 test in 2.282s 2022-08-17T13:37:10.6303578Z 2022-08-17T13:37:10.6303706Z OK 2022-08-17T13:37:10.6303795Z 2022-08-17T13:37:10.6303893Z Generating XML reports... 2022-08-17T13:37:10.6338611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133708.xml 2022-08-17T13:37:11.9292246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:11.9292678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:11.9295025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:11.9295597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:12.2727555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:12.2736730Z 2022-08-17T13:37:12.2736873Z Running tests... 2022-08-17T13:37:12.2737491Z ---------------------------------------------------------------------- 2022-08-17T13:37:12.5376127Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21062 2022-08-17T13:37:12.5404666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21063 2022-08-17T13:37:12.5432842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21064 2022-08-17T13:37:13.5284005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:13.5284672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:13.5286952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:13.5287564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:13.5320849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:13.5321496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:13.5325785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:13.5326880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:13.5427461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:13.5428132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:13.5431705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:13.5432376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:13.9189950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:13.9190716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:13.9191129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:13.9191785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:13.9192341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:13.9192889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:13.9297177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:13.9297927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:14.0202381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:14.0311703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:14.0412529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:14.0413328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:14.0414192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:14.0414753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:14.0415299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:14.0421966Z STAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0517692Z STAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0518577Z STAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0525407Z STAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0526150Z 2022-08-17T13:37:14.0526448Z STAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0534797Z STAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0535474Z STAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0536909Z STAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0544380Z STAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0544702Z 2022-08-17T13:37:14.0550674Z STAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0551086Z 2022-08-17T13:37:14.0552334Z STAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0558893Z STAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:14.0566664Z STAGE:2022-08-17 13:37:14 21064:21064 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0567862Z STAGE:2022-08-17 13:37:14 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:14 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:14.0568437Z 2022-08-17T13:37:14.5484358Z ok (2.274s) 2022-08-17T13:37:14.5484577Z 2022-08-17T13:37:14.5484901Z ---------------------------------------------------------------------- 2022-08-17T13:37:14.5485270Z Ran 1 test in 2.275s 2022-08-17T13:37:14.5485392Z 2022-08-17T13:37:14.5485456Z OK 2022-08-17T13:37:14.5485551Z 2022-08-17T13:37:14.5485645Z Generating XML reports... 2022-08-17T13:37:14.5522556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133712.xml 2022-08-17T13:37:15.8275330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:15.8275717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:15.8277908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:15.8278354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:16.1719849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:16.1729119Z 2022-08-17T13:37:16.1729243Z Running tests... 2022-08-17T13:37:16.1729611Z ---------------------------------------------------------------------- 2022-08-17T13:37:16.4341717Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21127 2022-08-17T13:37:16.4368469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21128 2022-08-17T13:37:16.4395922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21129 2022-08-17T13:37:17.4294007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:17.4294625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:17.4296904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:17.4297497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:17.4330375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:17.4331042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:17.4334715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:17.4335159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:17.4724045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:17.4724433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:17.4727646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:17.4728058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:17.8401794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:17.8503642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:17.8504378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:17.8505613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:17.8506253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:17.8506796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:17.8511697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:17.8512669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:17.8513954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:17.8514640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:17.8619986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:17.8620395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:17.8621194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:17.8621734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:17.8624253Z STAGE:2022-08-17 13:37:17 21128:21128 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:17.8717965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:17.8725575Z STAGE:2022-08-17 13:37:17 21129:21129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:17.8733076Z STAGE:2022-08-17 13:37:17 21128:21128 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:17 21129:21129 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:17.8733406Z 2022-08-17T13:37:17.8745329Z STAGE:2022-08-17 13:37:17 21129:21129 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:17 21128:21128 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:17.8745655Z 2022-08-17T13:37:17.8751012Z STAGE:2022-08-17 13:37:17 21128:21128 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:17 21129:21129 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:17.8751319Z 2022-08-17T13:37:18.3449497Z ok (2.172s) 2022-08-17T13:37:18.3449736Z 2022-08-17T13:37:18.3450307Z ---------------------------------------------------------------------- 2022-08-17T13:37:18.3450630Z Ran 1 test in 2.172s 2022-08-17T13:37:18.3450761Z 2022-08-17T13:37:18.3450811Z OK 2022-08-17T13:37:18.3450906Z 2022-08-17T13:37:18.3451001Z Generating XML reports... 2022-08-17T13:37:18.3484723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133716.xml 2022-08-17T13:37:19.6330700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:19.6331091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:19.6333287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:19.6333664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:19.9762290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:19.9772271Z 2022-08-17T13:37:19.9772452Z Running tests... 2022-08-17T13:37:19.9773046Z ---------------------------------------------------------------------- 2022-08-17T13:37:20.2428157Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21188 2022-08-17T13:37:20.2468434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21189 2022-08-17T13:37:20.2505672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21190 2022-08-17T13:37:21.2334131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:21.2334831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:21.2338460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:21.2339106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:21.2559238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:21.2559891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:21.2563530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:21.2563996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:21.2564848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:21.2565414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:21.2570642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:21.2571021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:21.6387920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:21.6462848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:21.6463285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:21.6463935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:21.6464494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:21.6489496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:21.6573574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:21.6574248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:21.6679251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:21.6679661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:21.7501119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:21.7503387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:21.7504431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:21.7590969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:21.7591822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:21.7595165Z STAGE:2022-08-17 13:37:21 21189:21189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:21.7694273Z STAGE:2022-08-17 13:37:21 21190:21190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:21.7700665Z STAGE:2022-08-17 13:37:21 21190:21190 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:21.7701408Z STAGE:2022-08-17 13:37:21 21189:21189 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:21.7710236Z STAGE:2022-08-17 13:37:21 21190:21190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:21.7711448Z STAGE:2022-08-17 13:37:21 21189:21189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:21.7716607Z STAGE:2022-08-17 13:37:21 21189:21189 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:21.7717362Z STAGE:2022-08-17 13:37:21 21190:21190 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:22.2555816Z ok (2.278s) 2022-08-17T13:37:22.2556057Z 2022-08-17T13:37:22.2556529Z ---------------------------------------------------------------------- 2022-08-17T13:37:22.2556971Z Ran 1 test in 2.278s 2022-08-17T13:37:22.2557151Z 2022-08-17T13:37:22.2557251Z OK 2022-08-17T13:37:22.2557396Z 2022-08-17T13:37:22.2557531Z Generating XML reports... 2022-08-17T13:37:22.2594067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133719.xml 2022-08-17T13:37:23.5538973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:23.5539409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:23.5541666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:23.5542193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:23.9042457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:23.9052817Z 2022-08-17T13:37:23.9052974Z Running tests... 2022-08-17T13:37:23.9053509Z ---------------------------------------------------------------------- 2022-08-17T13:37:24.1735521Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21249 2022-08-17T13:37:24.1763835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21250 2022-08-17T13:37:24.1793455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21251 2022-08-17T13:37:25.1610473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:25.1611140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:25.1613395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:25.1614043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:25.2059051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:25.2059711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:25.2062979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:25.2063625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:25.2212028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:25.2212744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:25.2215932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:25.2216601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:25.5936377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:25.6036961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:25.6037384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:25.6038027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:25.6038571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:25.6039143Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:25.6145052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:25.6145455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:25.6350681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:25.6351178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:25.7049873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:25.7051703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:25.7052395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:25.7059964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:25.7060600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:25.7063836Z STAGE:2022-08-17 13:37:25 21250:21250 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:25.7163696Z STAGE:2022-08-17 13:37:25 21251:21251 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:25.7169772Z STAGE:2022-08-17 13:37:25 21251:21251 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:25.7170256Z STAGE:2022-08-17 13:37:25 21250:21250 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:25.7179303Z STAGE:2022-08-17 13:37:25 21251:21251 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:25 21250:21250 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:25.7179954Z 2022-08-17T13:37:25.7184188Z STAGE:2022-08-17 13:37:25 21251:21251 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:25.7184955Z STAGE:2022-08-17 13:37:25 21250:21250 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:26.1846258Z ok (2.279s) 2022-08-17T13:37:26.1846397Z 2022-08-17T13:37:26.1846786Z ---------------------------------------------------------------------- 2022-08-17T13:37:26.1847060Z Ran 1 test in 2.279s 2022-08-17T13:37:26.1847181Z 2022-08-17T13:37:26.1847231Z OK 2022-08-17T13:37:26.1847325Z 2022-08-17T13:37:26.1847418Z Generating XML reports... 2022-08-17T13:37:26.1881601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133723.xml 2022-08-17T13:37:27.4819392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:27.4819883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:27.4822384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:27.4823096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:27.8282342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:27.8292244Z 2022-08-17T13:37:27.8292709Z Running tests... 2022-08-17T13:37:27.8293141Z ---------------------------------------------------------------------- 2022-08-17T13:37:28.0940443Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21310 2022-08-17T13:37:28.0968896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21311 2022-08-17T13:37:28.0996799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21312 2022-08-17T13:37:29.0922486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:29.0922919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:29.0926123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:29.0926801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:29.1044831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:29.1045494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:29.1048548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:29.1049205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:29.1363296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:29.1363989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:29.1367448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:29.1368100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:29.5010050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:29.5111416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:29.5112098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:29.5112943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:29.5113810Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:29.5114814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:29.5120332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:29.5121331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:29.5122083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:29.5122488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:37:29.5330698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:37:29.5331131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:37:29.5331903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:29.5332766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:29.5333790Z STAGE:2022-08-17 13:37:29 21311:21311 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:29.5334314Z STAGE:2022-08-17 13:37:29 21312:21312 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:29.5341122Z STAGE:2022-08-17 13:37:29 21312:21312 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:29.5341590Z STAGE:2022-08-17 13:37:29 21311:21311 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:29.5350517Z STAGE:2022-08-17 13:37:29 21312:21312 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:29 21311:21311 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:29.5351014Z 2022-08-17T13:37:29.5355667Z STAGE:2022-08-17 13:37:29 21311:21311 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:29 21312:21312 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:29.5356157Z 2022-08-17T13:37:29.5427373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:37:30.0044523Z ok (2.175s) 2022-08-17T13:37:30.0044775Z 2022-08-17T13:37:30.0045357Z ---------------------------------------------------------------------- 2022-08-17T13:37:30.0045678Z Ran 1 test in 2.175s 2022-08-17T13:37:30.0045803Z 2022-08-17T13:37:30.0046049Z OK 2022-08-17T13:37:30.0046150Z 2022-08-17T13:37:30.0046247Z Generating XML reports... 2022-08-17T13:37:30.0085123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133727.xml 2022-08-17T13:37:31.2696989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:31.2697379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:31.2700164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:31.2700529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:31.6108085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:31.6118557Z 2022-08-17T13:37:31.6118918Z Running tests... 2022-08-17T13:37:31.6119329Z ---------------------------------------------------------------------- 2022-08-17T13:37:31.8768085Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21371 2022-08-17T13:37:31.8795322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21372 2022-08-17T13:37:31.8823665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21373 2022-08-17T13:37:32.8530626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:32.8531269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:32.8533730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:32.8534380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:32.8663642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:32.8664306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:32.8667772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:32.8668199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:32.8716050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:32.8716757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:32.8720105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:32.8720486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:33.2240205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:33.2340213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:33.2340625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:33.2341411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:33.2342163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:33.2342802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:33.2448264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:33.2448972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:33.2452292Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.2452806Z STAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3352626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:33.3355871Z STAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3361999Z STAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3362675Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3363240Z STAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3372638Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3374962Z STAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3375453Z STAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3382412Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3383537Z STAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3383847Z 2022-08-17T13:37:33.3388624Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3390512Z STAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:33.3390948Z 2022-08-17T13:37:33.3397511Z STAGE:2022-08-17 13:37:33 21373:21373 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3398709Z STAGE:2022-08-17 13:37:33 21371:21371 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:33 21372:21372 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:33.3399254Z 2022-08-17T13:37:33.6872712Z ok (2.075s) 2022-08-17T13:37:33.6872987Z 2022-08-17T13:37:33.6873545Z ---------------------------------------------------------------------- 2022-08-17T13:37:33.6873913Z Ran 1 test in 2.075s 2022-08-17T13:37:33.6874043Z 2022-08-17T13:37:33.6874319Z OK 2022-08-17T13:37:33.6874406Z 2022-08-17T13:37:33.6874501Z Generating XML reports... 2022-08-17T13:37:33.6908136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133731.xml 2022-08-17T13:37:34.9729916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:34.9730630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:34.9733569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:34.9734193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:35.3105771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:35.3115613Z 2022-08-17T13:37:35.3116032Z Running tests... 2022-08-17T13:37:35.3116462Z ---------------------------------------------------------------------- 2022-08-17T13:37:35.5720326Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21427 2022-08-17T13:37:35.5748091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21428 2022-08-17T13:37:35.5776863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21429 2022-08-17T13:37:36.5694122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:36.5694581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:36.5695081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:36.5695495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:36.5696214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:36.5696764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:36.5698400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:36.5699046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:36.5699849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:36.5700443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:36.5701265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:36.5702209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:36.9522540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:36.9523268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:36.9523659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:36.9524310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:36.9524847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:36.9525404Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:36.9628875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:36.9632221Z STAGE:2022-08-17 13:37:36 21428:21428 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0534659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:37.0535376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:37.0538182Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0538853Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0545468Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0546021Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0546524Z STAGE:2022-08-17 13:37:37 21428:21428 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0555073Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0557599Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0558180Z STAGE:2022-08-17 13:37:37 21428:21428 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0566830Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:37 21428:21428 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0567395Z 2022-08-17T13:37:37.0570263Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0575343Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:37 21428:21428 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0575826Z 2022-08-17T13:37:37.0576775Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:37.0583613Z STAGE:2022-08-17 13:37:37 21427:21427 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0584391Z STAGE:2022-08-17 13:37:37 21429:21429 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.0585195Z STAGE:2022-08-17 13:37:37 21428:21428 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:37.4827066Z ok (2.171s) 2022-08-17T13:37:37.4827327Z 2022-08-17T13:37:37.4827779Z ---------------------------------------------------------------------- 2022-08-17T13:37:37.4828049Z Ran 1 test in 2.171s 2022-08-17T13:37:37.4828170Z 2022-08-17T13:37:37.4828233Z OK 2022-08-17T13:37:37.4828330Z 2022-08-17T13:37:37.4828424Z Generating XML reports... 2022-08-17T13:37:37.4863175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133735.xml 2022-08-17T13:37:38.7659765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:38.7660174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:38.7663210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:38.7663576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:39.1112258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:39.1121947Z 2022-08-17T13:37:39.1122063Z Running tests... 2022-08-17T13:37:39.1122662Z ---------------------------------------------------------------------- 2022-08-17T13:37:39.3776025Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21483 2022-08-17T13:37:39.3804847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21484 2022-08-17T13:37:39.3834396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21485 2022-08-17T13:37:40.3632565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:40.3633221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:40.3634884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:40.3635547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:40.3736738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:40.3737133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:40.3740126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:40.3740510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:40.3968031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:40.3968452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:40.3971818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:40.3972190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:40.7601593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:40.7703166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:40.7703931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:40.7704871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:40.7705437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:40.7705972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:40.7810378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:40.8716963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:40.8717676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:41.1885608Z skip: CUDA is not available. (2.076s) 2022-08-17T13:37:41.1886126Z 2022-08-17T13:37:41.1886873Z ---------------------------------------------------------------------- 2022-08-17T13:37:41.1887281Z Ran 1 test in 2.076s 2022-08-17T13:37:41.1887459Z 2022-08-17T13:37:41.1887585Z OK (skipped=1) 2022-08-17T13:37:41.1887749Z 2022-08-17T13:37:41.1887888Z Generating XML reports... 2022-08-17T13:37:41.1924980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133739.xml 2022-08-17T13:37:42.5071560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:42.5072262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:42.5075630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:42.5076258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:42.8549781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:42.8560127Z 2022-08-17T13:37:42.8560450Z Running tests... 2022-08-17T13:37:42.8561099Z ---------------------------------------------------------------------- 2022-08-17T13:37:43.1257515Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21536 2022-08-17T13:37:43.1287159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21537 2022-08-17T13:37:43.1316954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21538 2022-08-17T13:37:44.1232266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:44.1232950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:44.1235786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:44.1236403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:44.1247809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:44.1248469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:44.1252323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:44.1252981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:44.1465470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:44.1466162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:44.1468778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:44.1469183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:44.5062736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:44.5074654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:44.5075411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:44.5076047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:44.5076605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:44.5164368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:44.5183360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:44.5183948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:44.6177191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:44.9367809Z skip: CUDA is not available. (2.080s) 2022-08-17T13:37:44.9368148Z 2022-08-17T13:37:44.9368576Z ---------------------------------------------------------------------- 2022-08-17T13:37:44.9368850Z Ran 1 test in 2.081s 2022-08-17T13:37:44.9368974Z 2022-08-17T13:37:44.9369051Z OK (skipped=1) 2022-08-17T13:37:44.9369164Z 2022-08-17T13:37:44.9369256Z Generating XML reports... 2022-08-17T13:37:44.9403618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133742.xml 2022-08-17T13:37:46.2412643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:46.2413019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:46.2416382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:46.2417004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:46.5842495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:46.5853432Z 2022-08-17T13:37:46.5853747Z Running tests... 2022-08-17T13:37:46.5854419Z ---------------------------------------------------------------------- 2022-08-17T13:37:46.8463778Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21589 2022-08-17T13:37:46.8492101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21590 2022-08-17T13:37:46.8520206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21591 2022-08-17T13:37:47.8179925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:47.8180600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:47.8184143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:47.8184823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:47.8343279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:47.8343888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:47.8346401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:47.8347060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:47.8669591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:47.8670171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:47.8673346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:47.8673978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:48.2319129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:48.2418604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:48.2419022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:48.2419676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:48.2420429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:48.2421201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:48.2526572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:48.2527289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:48.2530043Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.2532236Z STAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3432266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:48.3436124Z STAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3443120Z STAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3443754Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3444208Z STAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3453323Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3456386Z STAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3456708Z 2022-08-17T13:37:48.3464180Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3464957Z STAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3465407Z STAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3470869Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3473426Z STAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:48.3473912Z 2022-08-17T13:37:48.3481199Z STAGE:2022-08-17 13:37:48 21590:21590 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3482381Z STAGE:2022-08-17 13:37:48 21591:21591 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:48 21589:21589 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:48.3482691Z 2022-08-17T13:37:48.7570141Z ok (2.171s) 2022-08-17T13:37:48.7570542Z 2022-08-17T13:37:48.7571044Z ---------------------------------------------------------------------- 2022-08-17T13:37:48.7571331Z Ran 1 test in 2.172s 2022-08-17T13:37:48.7571467Z 2022-08-17T13:37:48.7571518Z OK 2022-08-17T13:37:48.7571616Z 2022-08-17T13:37:48.7571710Z Generating XML reports... 2022-08-17T13:37:48.7609094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133746.xml 2022-08-17T13:37:50.0467270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:50.0467675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:50.0470205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:50.0470635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:50.3864307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:50.3873864Z 2022-08-17T13:37:50.3874100Z Running tests... 2022-08-17T13:37:50.3874571Z ---------------------------------------------------------------------- 2022-08-17T13:37:50.6506060Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21645 2022-08-17T13:37:50.6535365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21646 2022-08-17T13:37:50.6565284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21647 2022-08-17T13:37:51.6447888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:51.6448485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:51.6450604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:51.6451183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:51.6467408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:51.6468040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:51.6471287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:51.6471982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:51.7146713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:51.7147359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:51.7149879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:51.7150546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:52.0796596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:52.0897541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:52.0898202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:52.0899174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:52.0899737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:52.0900276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:52.1005208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:52.1912354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:52.1912773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:52.4615378Z skip: CUDA is not available. (2.074s) 2022-08-17T13:37:52.4615728Z 2022-08-17T13:37:52.4616274Z ---------------------------------------------------------------------- 2022-08-17T13:37:52.4616600Z Ran 1 test in 2.074s 2022-08-17T13:37:52.4616724Z 2022-08-17T13:37:52.4616816Z OK (skipped=1) 2022-08-17T13:37:52.4616918Z 2022-08-17T13:37:52.4617010Z Generating XML reports... 2022-08-17T13:37:52.4651260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133750.xml 2022-08-17T13:37:53.7585902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:53.7586307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:53.7588816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:53.7589443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:54.0976239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:54.0986450Z 2022-08-17T13:37:54.0986715Z Running tests... 2022-08-17T13:37:54.0987097Z ---------------------------------------------------------------------- 2022-08-17T13:37:54.3620953Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21698 2022-08-17T13:37:54.3649609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21699 2022-08-17T13:37:54.3680704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21700 2022-08-17T13:37:55.3421809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:55.3422435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:55.3424778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:55.3425456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:55.3453616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:55.3454271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:55.3457507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:55.3457932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:55.3575857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:55.3576499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:55.3579635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:55.3580306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:55.7206375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:55.7206976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:55.7207372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:55.7208063Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:55.7208704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:55.7209337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:55.7312160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:55.7315146Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8219295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:55.8219895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:55.8223252Z STAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8224002Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8230533Z STAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8231026Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8231492Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8240723Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8241073Z 2022-08-17T13:37:55.8243689Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8251402Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8252102Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8252415Z 2022-08-17T13:37:55.8258372Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8259614Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8260081Z STAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:55.8268110Z STAGE:2022-08-17 13:37:55 21698:21698 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8268599Z STAGE:2022-08-17 13:37:55 21699:21699 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:55.8274723Z STAGE:2022-08-17 13:37:55 21700:21700 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:56.2732417Z ok (2.174s) 2022-08-17T13:37:56.2732576Z 2022-08-17T13:37:56.2733062Z ---------------------------------------------------------------------- 2022-08-17T13:37:56.2733334Z Ran 1 test in 2.175s 2022-08-17T13:37:56.2733461Z 2022-08-17T13:37:56.2733526Z OK 2022-08-17T13:37:56.2733612Z 2022-08-17T13:37:56.2733703Z Generating XML reports... 2022-08-17T13:37:56.2771460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133754.xml 2022-08-17T13:37:57.5703453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:57.5703854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:57.5706964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:57.5707396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:57.9107580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:37:57.9117104Z 2022-08-17T13:37:57.9117255Z Running tests... 2022-08-17T13:37:57.9117700Z ---------------------------------------------------------------------- 2022-08-17T13:37:58.1743886Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21754 2022-08-17T13:37:58.1773852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21755 2022-08-17T13:37:58.1801410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21756 2022-08-17T13:37:59.2305729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:59.2306127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:59.2308867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:59.2309255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:59.2369042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:59.2373128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:59.2374246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:59.2374905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:59.2489395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:37:59.2489995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:37:59.2493768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:37:59.2494453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:37:59.6165744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:37:59.6266786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:37:59.6267496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:37:59.6268503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:59.6269476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:59.6270159Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:37:59.6276789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:37:59.6277480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:37:59.6278485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:37:59.6280947Z STAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6281592Z STAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6282769Z STAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6289600Z STAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6290145Z STAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6290622Z STAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6299490Z STAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6300234Z STAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6300526Z 2022-08-17T13:37:59.6307113Z STAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6307566Z 2022-08-17T13:37:59.6307582Z 2022-08-17T13:37:59.6313604Z STAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6314089Z STAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6314635Z STAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:37:59.6322086Z STAGE:2022-08-17 13:37:59 21755:21755 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6322910Z STAGE:2022-08-17 13:37:59 21754:21754 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:37:59 21756:21756 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:37:59.6323397Z 2022-08-17T13:37:59.9852981Z ok (2.073s) 2022-08-17T13:37:59.9853246Z 2022-08-17T13:37:59.9853770Z ---------------------------------------------------------------------- 2022-08-17T13:37:59.9854198Z Ran 1 test in 2.074s 2022-08-17T13:37:59.9854321Z 2022-08-17T13:37:59.9854373Z OK 2022-08-17T13:37:59.9854468Z 2022-08-17T13:37:59.9854562Z Generating XML reports... 2022-08-17T13:37:59.9888758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133757.xml 2022-08-17T13:38:01.2409936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:01.2410359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:01.2412224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:01.2412619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:01.5828919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:01.5838948Z 2022-08-17T13:38:01.5839054Z Running tests... 2022-08-17T13:38:01.5839753Z ---------------------------------------------------------------------- 2022-08-17T13:38:01.8510331Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21810 2022-08-17T13:38:01.8537815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21811 2022-08-17T13:38:01.8565599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21812 2022-08-17T13:38:02.8612180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:02.8612850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:02.8616064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:02.8616681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:02.8745635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:02.8746017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:02.8749680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:02.8750151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:02.8853762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:02.8856914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:02.8857471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:02.8857847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:03.2529194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:38:03.2630088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:38:03.2630649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:38:03.2631315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:03.2631882Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:03.2632447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:03.2737816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:38:03.2741864Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3642900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:38:03.3643309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:38:03.3647631Z STAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3648357Z STAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3656156Z STAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.3657293Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.3657795Z 2022-08-17T13:38:03.3666875Z STAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3667795Z STAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3668527Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3675242Z STAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.3675765Z 2022-08-17T13:38:03.3676180Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.3682033Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3682368Z 2022-08-17T13:38:03.3683427Z STAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:38:03.3690260Z STAGE:2022-08-17 13:38:03 21810:21810 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:38:03 21811:21811 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.3690793Z 2022-08-17T13:38:03.3691252Z STAGE:2022-08-17 13:38:03 21812:21812 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:38:03.7618556Z ok (2.178s) 2022-08-17T13:38:03.7618824Z 2022-08-17T13:38:03.7619373Z ---------------------------------------------------------------------- 2022-08-17T13:38:03.7619696Z Ran 1 test in 2.178s 2022-08-17T13:38:03.7619821Z 2022-08-17T13:38:03.7619885Z OK 2022-08-17T13:38:03.7619981Z 2022-08-17T13:38:03.7620075Z Generating XML reports... 2022-08-17T13:38:03.7653957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133801.xml 2022-08-17T13:38:05.0714958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:05.0715337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:05.0718092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:05.0718466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:05.4159127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:05.4169284Z 2022-08-17T13:38:05.4169425Z Running tests... 2022-08-17T13:38:05.4169964Z ---------------------------------------------------------------------- 2022-08-17T13:38:05.6801309Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21866 2022-08-17T13:38:05.6831731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21867 2022-08-17T13:38:05.6862942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21868 2022-08-17T13:38:06.6669746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:06.6670124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:06.6673001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:06.6673377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:06.6869746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:06.6870212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:06.6870668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:06.6871011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:06.6874544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:06.6875198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:06.6876047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:06.6876663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:07.0771637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:38:07.0872258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:38:07.0872880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:38:07.0873538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:07.0874107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:07.0874660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:07.0979106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:38:07.1885640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:38:07.1886405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:38:07.4909174Z skip: CUDA is not available. (2.074s) 2022-08-17T13:38:07.4909537Z 2022-08-17T13:38:07.4909958Z ---------------------------------------------------------------------- 2022-08-17T13:38:07.4910209Z Ran 1 test in 2.074s 2022-08-17T13:38:07.4910332Z 2022-08-17T13:38:07.4910409Z OK (skipped=1) 2022-08-17T13:38:07.4910522Z 2022-08-17T13:38:07.4910619Z Generating XML reports... 2022-08-17T13:38:07.4945562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133805.xml 2022-08-17T13:38:08.7602300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:08.7602726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:08.7603959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:08.7604517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:09.1009927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:09.1019471Z 2022-08-17T13:38:09.1019961Z Running tests... 2022-08-17T13:38:09.1020592Z ---------------------------------------------------------------------- 2022-08-17T13:38:09.3672551Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21919 2022-08-17T13:38:09.3700869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21920 2022-08-17T13:38:09.3729827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21921 2022-08-17T13:38:10.4356893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:10.4357542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:10.4359273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:10.4359905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:10.4473852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:10.4474726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:10.4477786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:10.4478191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:10.4722608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:10.4723156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:10.4726421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:10.4727096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:10.8456971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:38:10.8558606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:38:10.8559195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:38:10.8559995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:10.8560909Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:10.8561636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:10.8567293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:38:10.8568176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:38:10.8569081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:38:11.1776968Z skip: CUDA is not available. (2.075s) 2022-08-17T13:38:11.1777179Z 2022-08-17T13:38:11.1777509Z ---------------------------------------------------------------------- 2022-08-17T13:38:11.1777771Z Ran 1 test in 2.076s 2022-08-17T13:38:11.1777877Z 2022-08-17T13:38:11.1777954Z OK (skipped=1) 2022-08-17T13:38:11.1778067Z 2022-08-17T13:38:11.1778161Z Generating XML reports... 2022-08-17T13:38:11.1812544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133809.xml 2022-08-17T13:38:12.4390179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:12.4390849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:12.4393383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:12.4393800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:12.7831636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:12.7842178Z 2022-08-17T13:38:12.7842458Z Running tests... 2022-08-17T13:38:12.7843111Z ---------------------------------------------------------------------- 2022-08-17T13:38:13.0559348Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21972 2022-08-17T13:38:13.0587940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21973 2022-08-17T13:38:13.0615829Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21974 2022-08-17T13:38:14.0347838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:14.0348453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:14.0350819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:14.0351436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:14.0533978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:14.0536563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:14.0537435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:14.0538051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:14.0818486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:14.0819069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:14.0821577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:14.0822126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:14.4443344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:38:14.4481431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:38:14.4481886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:38:14.4482581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:14.4483216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:14.4545017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:14.4589968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:38:14.4590470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:38:14.5558522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:38:14.8663854Z skip: CUDA is not available. (2.082s) 2022-08-17T13:38:14.8664119Z 2022-08-17T13:38:14.8664587Z ---------------------------------------------------------------------- 2022-08-17T13:38:14.8665003Z Ran 1 test in 2.082s 2022-08-17T13:38:14.8665197Z 2022-08-17T13:38:14.8665310Z OK (skipped=1) 2022-08-17T13:38:14.8667051Z 2022-08-17T13:38:14.8667293Z Generating XML reports... 2022-08-17T13:38:14.8701556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133812.xml 2022-08-17T13:38:16.1813139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:16.1813561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:16.1815747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:16.1816130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:16.5282815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:16.5292852Z 2022-08-17T13:38:16.5292962Z Running tests... 2022-08-17T13:38:16.5293431Z ---------------------------------------------------------------------- 2022-08-17T13:38:16.5310867Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:38:16.5311421Z 2022-08-17T13:38:16.5311952Z ---------------------------------------------------------------------- 2022-08-17T13:38:16.5312235Z Ran 1 test in 0.002s 2022-08-17T13:38:16.5312357Z 2022-08-17T13:38:16.5312421Z OK (skipped=1) 2022-08-17T13:38:16.5312532Z 2022-08-17T13:38:16.5312845Z Generating XML reports... 2022-08-17T13:38:16.5337581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133816.xml 2022-08-17T13:38:17.7065687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:17.7066765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:17.7091225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:17.7091682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:18.0461620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:18.0471020Z 2022-08-17T13:38:18.0471130Z Running tests... 2022-08-17T13:38:18.0471754Z ---------------------------------------------------------------------- 2022-08-17T13:38:18.0489235Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:38:18.0489466Z 2022-08-17T13:38:18.0489741Z ---------------------------------------------------------------------- 2022-08-17T13:38:18.0491101Z Ran 1 test in 0.002s 2022-08-17T13:38:18.0491310Z 2022-08-17T13:38:18.0491420Z OK (skipped=1) 2022-08-17T13:38:18.0491541Z 2022-08-17T13:38:18.0491635Z Generating XML reports... 2022-08-17T13:38:18.0515296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133818.xml 2022-08-17T13:38:19.2060748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:19.2061414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:19.2064528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:19.2065128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:19.5528796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:19.5539278Z 2022-08-17T13:38:19.5539761Z Running tests... 2022-08-17T13:38:19.5540197Z ---------------------------------------------------------------------- 2022-08-17T13:38:19.5558165Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:38:19.5558593Z 2022-08-17T13:38:19.5559088Z ---------------------------------------------------------------------- 2022-08-17T13:38:19.5559391Z Ran 1 test in 0.002s 2022-08-17T13:38:19.5559769Z 2022-08-17T13:38:19.5559844Z OK (skipped=1) 2022-08-17T13:38:19.5559958Z 2022-08-17T13:38:19.5560047Z Generating XML reports... 2022-08-17T13:38:19.5584853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133819.xml 2022-08-17T13:38:20.7172297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:20.7172694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:20.7174907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:20.7175465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:21.0592740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:21.0601763Z 2022-08-17T13:38:21.0602048Z Running tests... 2022-08-17T13:38:21.0602639Z ---------------------------------------------------------------------- 2022-08-17T13:38:21.0620637Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:38:21.0620968Z 2022-08-17T13:38:21.0621330Z ---------------------------------------------------------------------- 2022-08-17T13:38:21.0621830Z Ran 1 test in 0.002s 2022-08-17T13:38:21.0621951Z 2022-08-17T13:38:21.0622032Z OK (skipped=1) 2022-08-17T13:38:21.0622144Z 2022-08-17T13:38:21.0622232Z Generating XML reports... 2022-08-17T13:38:21.0646721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133821.xml 2022-08-17T13:38:22.2039888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:22.2040288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:22.2042872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:22.2043275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:22.5447585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:22.5457681Z 2022-08-17T13:38:22.5457912Z Running tests... 2022-08-17T13:38:22.5458293Z ---------------------------------------------------------------------- 2022-08-17T13:38:22.5474845Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:38:22.5475239Z 2022-08-17T13:38:22.5475644Z ---------------------------------------------------------------------- 2022-08-17T13:38:22.5476099Z Ran 1 test in 0.002s 2022-08-17T13:38:22.5476229Z 2022-08-17T13:38:22.5476309Z OK (skipped=1) 2022-08-17T13:38:22.5476421Z 2022-08-17T13:38:22.5476496Z Generating XML reports... 2022-08-17T13:38:22.5501198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133822.xml 2022-08-17T13:38:23.7033581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:23.7033983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:23.7036230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:23.7036603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:24.0472279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:24.0481555Z 2022-08-17T13:38:24.0481664Z Running tests... 2022-08-17T13:38:24.0482133Z ---------------------------------------------------------------------- 2022-08-17T13:38:24.0500419Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-08-17T13:38:24.0501171Z 2022-08-17T13:38:24.0501452Z ---------------------------------------------------------------------- 2022-08-17T13:38:24.0501703Z Ran 1 test in 0.002s 2022-08-17T13:38:24.0501824Z 2022-08-17T13:38:24.0501901Z OK (skipped=1) 2022-08-17T13:38:24.0502017Z 2022-08-17T13:38:24.0502107Z Generating XML reports... 2022-08-17T13:38:24.0526351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133824.xml 2022-08-17T13:38:25.1956742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:25.1957132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:25.1959264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:25.1959790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:25.5401415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:25.5411850Z 2022-08-17T13:38:25.5412050Z Running tests... 2022-08-17T13:38:25.5412406Z ---------------------------------------------------------------------- 2022-08-17T13:38:25.5429964Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-08-17T13:38:25.5430274Z 2022-08-17T13:38:25.5430633Z ---------------------------------------------------------------------- 2022-08-17T13:38:25.5430880Z Ran 1 test in 0.002s 2022-08-17T13:38:25.5430998Z 2022-08-17T13:38:25.5431076Z OK (skipped=1) 2022-08-17T13:38:25.5431190Z 2022-08-17T13:38:25.5431310Z Generating XML reports... 2022-08-17T13:38:25.5456082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133825.xml 2022-08-17T13:38:26.7009634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:26.7010067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:26.7012155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:26.7012557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:27.0473537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:27.0484822Z 2022-08-17T13:38:27.0484964Z Running tests... 2022-08-17T13:38:27.0485373Z ---------------------------------------------------------------------- 2022-08-17T13:38:27.0503684Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:27.0504181Z 2022-08-17T13:38:27.0504811Z ---------------------------------------------------------------------- 2022-08-17T13:38:27.0505264Z Ran 1 test in 0.002s 2022-08-17T13:38:27.0505458Z 2022-08-17T13:38:27.0505541Z OK (skipped=1) 2022-08-17T13:38:27.0505653Z 2022-08-17T13:38:27.0505743Z Generating XML reports... 2022-08-17T13:38:27.0530069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133827.xml 2022-08-17T13:38:28.2124442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:28.2125163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:28.2128646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:28.2129039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:28.5640610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:28.5649555Z 2022-08-17T13:38:28.5649957Z Running tests... 2022-08-17T13:38:28.5650366Z ---------------------------------------------------------------------- 2022-08-17T13:38:28.5667358Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:28.5667664Z 2022-08-17T13:38:28.5668047Z ---------------------------------------------------------------------- 2022-08-17T13:38:28.5668521Z Ran 1 test in 0.002s 2022-08-17T13:38:28.5668725Z 2022-08-17T13:38:28.5668790Z OK (skipped=1) 2022-08-17T13:38:28.5668904Z 2022-08-17T13:38:28.5668996Z Generating XML reports... 2022-08-17T13:38:28.5693802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133828.xml 2022-08-17T13:38:29.7310983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:29.7311365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:29.7313664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:29.7314052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:30.0816234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:30.0826396Z 2022-08-17T13:38:30.0826520Z Running tests... 2022-08-17T13:38:30.0827428Z ---------------------------------------------------------------------- 2022-08-17T13:38:30.0844304Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:30.0844760Z 2022-08-17T13:38:30.0845186Z ---------------------------------------------------------------------- 2022-08-17T13:38:30.0845613Z Ran 1 test in 0.002s 2022-08-17T13:38:30.0845805Z 2022-08-17T13:38:30.0846207Z OK (skipped=1) 2022-08-17T13:38:30.0846406Z 2022-08-17T13:38:30.0846548Z Generating XML reports... 2022-08-17T13:38:30.0871806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133830.xml 2022-08-17T13:38:31.2430510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:31.2431113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:31.2433369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:31.2433983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:31.5849365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:31.5859734Z 2022-08-17T13:38:31.5860207Z Running tests... 2022-08-17T13:38:31.5860640Z ---------------------------------------------------------------------- 2022-08-17T13:38:31.5878161Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:31.5878533Z 2022-08-17T13:38:31.5878956Z ---------------------------------------------------------------------- 2022-08-17T13:38:31.5879342Z Ran 1 test in 0.002s 2022-08-17T13:38:31.5879462Z 2022-08-17T13:38:31.5879534Z OK (skipped=1) 2022-08-17T13:38:31.5879633Z 2022-08-17T13:38:31.5879720Z Generating XML reports... 2022-08-17T13:38:31.5904423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133831.xml 2022-08-17T13:38:32.7351628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:32.7352283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:32.7354880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:32.7355474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:33.0777706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:33.0788459Z 2022-08-17T13:38:33.0788775Z Running tests... 2022-08-17T13:38:33.0789422Z ---------------------------------------------------------------------- 2022-08-17T13:38:33.0808321Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:33.0808851Z 2022-08-17T13:38:33.0809149Z ---------------------------------------------------------------------- 2022-08-17T13:38:33.0809410Z Ran 1 test in 0.002s 2022-08-17T13:38:33.0809531Z 2022-08-17T13:38:33.0809607Z OK (skipped=1) 2022-08-17T13:38:33.0809706Z 2022-08-17T13:38:33.0809796Z Generating XML reports... 2022-08-17T13:38:33.0834348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133833.xml 2022-08-17T13:38:34.2497745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:34.2498157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:34.2500884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:34.2501480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:34.5941851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:34.5951878Z 2022-08-17T13:38:34.5952135Z Running tests... 2022-08-17T13:38:34.5952590Z ---------------------------------------------------------------------- 2022-08-17T13:38:34.5969480Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:34.5969765Z 2022-08-17T13:38:34.5970118Z ---------------------------------------------------------------------- 2022-08-17T13:38:34.5970386Z Ran 1 test in 0.002s 2022-08-17T13:38:34.5970507Z 2022-08-17T13:38:34.5970582Z OK (skipped=1) 2022-08-17T13:38:34.5970693Z 2022-08-17T13:38:34.5970783Z Generating XML reports... 2022-08-17T13:38:34.5995809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133834.xml 2022-08-17T13:38:35.7582658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:35.7583059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:35.7584966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:35.7585346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:36.1045291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:36.1055611Z 2022-08-17T13:38:36.1055711Z Running tests... 2022-08-17T13:38:36.1056637Z ---------------------------------------------------------------------- 2022-08-17T13:38:36.1075069Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:36.1075585Z 2022-08-17T13:38:36.1076077Z ---------------------------------------------------------------------- 2022-08-17T13:38:36.1076493Z Ran 1 test in 0.002s 2022-08-17T13:38:36.1076616Z 2022-08-17T13:38:36.1076691Z OK (skipped=1) 2022-08-17T13:38:36.1076806Z 2022-08-17T13:38:36.1076901Z Generating XML reports... 2022-08-17T13:38:36.1101551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133836.xml 2022-08-17T13:38:37.2745543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:37.2745932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:37.2748528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:37.2749266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:37.6179257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:37.6188299Z 2022-08-17T13:38:37.6188593Z Running tests... 2022-08-17T13:38:37.6189208Z ---------------------------------------------------------------------- 2022-08-17T13:38:37.6207343Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:37.6208479Z 2022-08-17T13:38:37.6208824Z ---------------------------------------------------------------------- 2022-08-17T13:38:37.6209310Z Ran 1 test in 0.002s 2022-08-17T13:38:37.6209516Z 2022-08-17T13:38:37.6209597Z OK (skipped=1) 2022-08-17T13:38:37.6209717Z 2022-08-17T13:38:37.6209792Z Generating XML reports... 2022-08-17T13:38:37.6233943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133837.xml 2022-08-17T13:38:38.7843368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:38.7844009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:38.7846606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:38.7846984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:39.1296621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:39.1306491Z 2022-08-17T13:38:39.1306847Z Running tests... 2022-08-17T13:38:39.1307270Z ---------------------------------------------------------------------- 2022-08-17T13:38:39.1325060Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:39.1325426Z 2022-08-17T13:38:39.1325696Z ---------------------------------------------------------------------- 2022-08-17T13:38:39.1326122Z Ran 1 test in 0.002s 2022-08-17T13:38:39.1326243Z 2022-08-17T13:38:39.1326321Z OK (skipped=1) 2022-08-17T13:38:39.1326445Z 2022-08-17T13:38:39.1326534Z Generating XML reports... 2022-08-17T13:38:39.1351455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133839.xml 2022-08-17T13:38:40.2916425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:40.2917129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:40.2921032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:40.2921656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:40.6349620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:40.6358236Z 2022-08-17T13:38:40.6358336Z Running tests... 2022-08-17T13:38:40.6359025Z ---------------------------------------------------------------------- 2022-08-17T13:38:40.6376137Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:40.6376499Z 2022-08-17T13:38:40.6376985Z ---------------------------------------------------------------------- 2022-08-17T13:38:40.6377450Z Ran 1 test in 0.002s 2022-08-17T13:38:40.6377646Z 2022-08-17T13:38:40.6377770Z OK (skipped=1) 2022-08-17T13:38:40.6377948Z 2022-08-17T13:38:40.6378094Z Generating XML reports... 2022-08-17T13:38:40.6403488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133840.xml 2022-08-17T13:38:41.8054376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:41.8055021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:41.8057564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:41.8058152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:42.1467043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:42.1477936Z 2022-08-17T13:38:42.1478240Z Running tests... 2022-08-17T13:38:42.1478929Z ---------------------------------------------------------------------- 2022-08-17T13:38:42.1497055Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:42.1497488Z 2022-08-17T13:38:42.1497915Z ---------------------------------------------------------------------- 2022-08-17T13:38:42.1498350Z Ran 1 test in 0.002s 2022-08-17T13:38:42.1498562Z 2022-08-17T13:38:42.1498694Z OK (skipped=1) 2022-08-17T13:38:42.1498898Z 2022-08-17T13:38:42.1499060Z Generating XML reports... 2022-08-17T13:38:42.1524479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133842.xml 2022-08-17T13:38:43.3017146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:43.3017824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:43.3020412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:43.3021002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:43.6479880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:43.6490537Z 2022-08-17T13:38:43.6490870Z Running tests... 2022-08-17T13:38:43.6491349Z ---------------------------------------------------------------------- 2022-08-17T13:38:43.6509329Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:43.6509799Z 2022-08-17T13:38:43.6510293Z ---------------------------------------------------------------------- 2022-08-17T13:38:43.6510609Z Ran 1 test in 0.002s 2022-08-17T13:38:43.6510730Z 2022-08-17T13:38:43.6510795Z OK (skipped=1) 2022-08-17T13:38:43.6510908Z 2022-08-17T13:38:43.6511005Z Generating XML reports... 2022-08-17T13:38:43.6535759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133843.xml 2022-08-17T13:38:44.8031083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:44.8031474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:44.8034057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:44.8034514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:45.1488735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:45.1498581Z 2022-08-17T13:38:45.1498723Z Running tests... 2022-08-17T13:38:45.1499324Z ---------------------------------------------------------------------- 2022-08-17T13:38:45.1518260Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:45.1518804Z 2022-08-17T13:38:45.1519121Z ---------------------------------------------------------------------- 2022-08-17T13:38:45.1519381Z Ran 1 test in 0.002s 2022-08-17T13:38:45.1519504Z 2022-08-17T13:38:45.1519567Z OK (skipped=1) 2022-08-17T13:38:45.1519932Z 2022-08-17T13:38:45.1520020Z Generating XML reports... 2022-08-17T13:38:45.1544410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133845.xml 2022-08-17T13:38:46.2938314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:46.2938713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:46.2942576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:46.2942988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:46.6369090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:46.6378554Z 2022-08-17T13:38:46.6378803Z Running tests... 2022-08-17T13:38:46.6379433Z ---------------------------------------------------------------------- 2022-08-17T13:38:46.6396183Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:46.6396709Z 2022-08-17T13:38:46.6397009Z ---------------------------------------------------------------------- 2022-08-17T13:38:46.6397254Z Ran 1 test in 0.002s 2022-08-17T13:38:46.6397608Z 2022-08-17T13:38:46.6397687Z OK (skipped=1) 2022-08-17T13:38:46.6397802Z 2022-08-17T13:38:46.6397891Z Generating XML reports... 2022-08-17T13:38:46.6422485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133846.xml 2022-08-17T13:38:47.7842559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:47.7843201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:47.7846152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:47.7846788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:48.1281671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:48.1292238Z 2022-08-17T13:38:48.1292759Z Running tests... 2022-08-17T13:38:48.1293210Z ---------------------------------------------------------------------- 2022-08-17T13:38:48.1310551Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:48.1311013Z 2022-08-17T13:38:48.1311440Z ---------------------------------------------------------------------- 2022-08-17T13:38:48.1311766Z Ran 1 test in 0.002s 2022-08-17T13:38:48.1311890Z 2022-08-17T13:38:48.1311964Z OK (skipped=1) 2022-08-17T13:38:48.1312075Z 2022-08-17T13:38:48.1312169Z Generating XML reports... 2022-08-17T13:38:48.1336918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133848.xml 2022-08-17T13:38:49.2849741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:49.2850145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:49.2852346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:49.2852722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:49.6318916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:49.6329241Z 2022-08-17T13:38:49.6329670Z Running tests... 2022-08-17T13:38:49.6330085Z ---------------------------------------------------------------------- 2022-08-17T13:38:49.6347288Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-08-17T13:38:49.6348047Z 2022-08-17T13:38:49.6348460Z ---------------------------------------------------------------------- 2022-08-17T13:38:49.6348871Z Ran 1 test in 0.002s 2022-08-17T13:38:49.6349073Z 2022-08-17T13:38:49.6349204Z OK (skipped=1) 2022-08-17T13:38:49.6349400Z 2022-08-17T13:38:49.6349559Z Generating XML reports... 2022-08-17T13:38:49.6374536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133849.xml 2022-08-17T13:38:50.8000357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:50.8000996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:50.8003494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:50.8004074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:51.1426735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:51.1436877Z 2022-08-17T13:38:51.1437308Z Running tests... 2022-08-17T13:38:51.1437947Z ---------------------------------------------------------------------- 2022-08-17T13:38:51.1456331Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-08-17T13:38:51.1456643Z 2022-08-17T13:38:51.1456981Z ---------------------------------------------------------------------- 2022-08-17T13:38:51.1457249Z Ran 1 test in 0.002s 2022-08-17T13:38:51.1457370Z 2022-08-17T13:38:51.1457432Z OK (skipped=1) 2022-08-17T13:38:51.1457544Z 2022-08-17T13:38:51.1457634Z Generating XML reports... 2022-08-17T13:38:51.1481812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133851.xml 2022-08-17T13:38:52.3122968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:52.3123363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:52.3125568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:52.3126328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:52.6553702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:52.6563279Z 2022-08-17T13:38:52.6563429Z Running tests... 2022-08-17T13:38:52.6564059Z ---------------------------------------------------------------------- 2022-08-17T13:38:52.9224862Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22265 2022-08-17T13:38:52.9253892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22266 2022-08-17T13:38:52.9282428Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22267 2022-08-17T13:38:53.9091993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:53.9092691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:53.9096603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:53.9097240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:53.9133436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:53.9133809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:53.9137244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:53.9137693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:53.9170643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:53.9171037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:53.9175178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:53.9175561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:54.2853888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:38:54.2955012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:38:54.2955436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:38:54.2956078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:54.2956653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:54.2957203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:38:54.2963336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:38:54.2964139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:38:54.2964831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:38:54.6328267Z skip: Need at least 2 CUDA devices (1.976s) 2022-08-17T13:38:54.6328597Z 2022-08-17T13:38:54.6329148Z ---------------------------------------------------------------------- 2022-08-17T13:38:54.6329519Z Ran 1 test in 1.976s 2022-08-17T13:38:54.6329641Z 2022-08-17T13:38:54.6329719Z OK (skipped=1) 2022-08-17T13:38:54.6329836Z 2022-08-17T13:38:54.6329943Z Generating XML reports... 2022-08-17T13:38:54.6364416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133852.xml 2022-08-17T13:38:55.9220766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:55.9221175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:55.9223927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:55.9224296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:56.2607748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:56.2618181Z 2022-08-17T13:38:56.2618519Z Running tests... 2022-08-17T13:38:56.2619180Z ---------------------------------------------------------------------- 2022-08-17T13:38:56.2637158Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:38:56.2637622Z 2022-08-17T13:38:56.2637968Z ---------------------------------------------------------------------- 2022-08-17T13:38:56.2638395Z Ran 1 test in 0.002s 2022-08-17T13:38:56.2638603Z 2022-08-17T13:38:56.2638741Z OK (skipped=1) 2022-08-17T13:38:56.2638944Z 2022-08-17T13:38:56.2639102Z Generating XML reports... 2022-08-17T13:38:56.2664389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133856.xml 2022-08-17T13:38:57.4134387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:57.4135059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:57.4137672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:57.4138577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:57.7589456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:57.7599686Z 2022-08-17T13:38:57.7600002Z Running tests... 2022-08-17T13:38:57.7600672Z ---------------------------------------------------------------------- 2022-08-17T13:38:57.7618278Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:38:57.7618741Z 2022-08-17T13:38:57.7619569Z ---------------------------------------------------------------------- 2022-08-17T13:38:57.7620017Z Ran 1 test in 0.002s 2022-08-17T13:38:57.7620142Z 2022-08-17T13:38:57.7620211Z OK (skipped=1) 2022-08-17T13:38:57.7620334Z 2022-08-17T13:38:57.7620425Z Generating XML reports... 2022-08-17T13:38:57.7644434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133857.xml 2022-08-17T13:38:58.9350680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:38:58.9351348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:38:58.9354497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:38:58.9355100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:38:59.2768228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:38:59.2778433Z 2022-08-17T13:38:59.2778882Z Running tests... 2022-08-17T13:38:59.2779283Z ---------------------------------------------------------------------- 2022-08-17T13:38:59.5402215Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22338 2022-08-17T13:38:59.5431557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22339 2022-08-17T13:38:59.5460106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22340 2022-08-17T13:39:00.5479251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:00.5479865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:00.5482499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:00.5483094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:00.5592401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:00.5592994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:00.5596696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:00.5597349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:00.5967539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:00.5968219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:00.5971572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:00.5972151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:00.9559542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:00.9660383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:00.9660911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:00.9661714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:00.9662551Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:00.9663116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:00.9669478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:00.9670410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:00.9671017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:02.5525786Z ok (3.274s) 2022-08-17T13:39:02.5526474Z 2022-08-17T13:39:02.5526925Z ---------------------------------------------------------------------- 2022-08-17T13:39:02.5527179Z Ran 1 test in 3.275s 2022-08-17T13:39:02.5527320Z 2022-08-17T13:39:02.5527386Z OK 2022-08-17T13:39:02.5527483Z 2022-08-17T13:39:02.5527579Z Generating XML reports... 2022-08-17T13:39:02.5561915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133859.xml 2022-08-17T13:39:03.8428277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:03.8428676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:03.8431314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:03.8431704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:04.1899392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:04.1909091Z 2022-08-17T13:39:04.1909244Z Running tests... 2022-08-17T13:39:04.1909884Z ---------------------------------------------------------------------- 2022-08-17T13:39:04.4596937Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22391 2022-08-17T13:39:04.4625951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22392 2022-08-17T13:39:04.4654766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22393 2022-08-17T13:39:05.4564919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:05.4568031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:05.4568903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:05.4569523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:05.4623627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:05.4624260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:05.4628601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:05.4629276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:05.4812557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:05.4813270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:05.4817082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:05.4817722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:05.8632963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:05.8633583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:05.8634150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:05.8634988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:05.8635976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:05.8636846Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:05.8641187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:05.8641806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:05.8642479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:06.1700591Z skip: CUDA is not available. (1.979s) 2022-08-17T13:39:06.1700940Z 2022-08-17T13:39:06.1701432Z ---------------------------------------------------------------------- 2022-08-17T13:39:06.1701681Z Ran 1 test in 1.979s 2022-08-17T13:39:06.1701800Z 2022-08-17T13:39:06.1701876Z OK (skipped=1) 2022-08-17T13:39:06.1702205Z 2022-08-17T13:39:06.1702373Z Generating XML reports... 2022-08-17T13:39:06.1736434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133904.xml 2022-08-17T13:39:07.4489712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:07.4490100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:07.4492211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:07.4492714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:07.7912950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:07.7922558Z 2022-08-17T13:39:07.7922689Z Running tests... 2022-08-17T13:39:07.7923336Z ---------------------------------------------------------------------- 2022-08-17T13:39:08.0565104Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22444 2022-08-17T13:39:08.0593120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22445 2022-08-17T13:39:08.0621243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22446 2022-08-17T13:39:09.0510966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:09.0511652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:09.0513784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:09.0514444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:09.0869827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:09.0870572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:09.0873148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:09.0873745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:09.0908001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:09.0908403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:09.0912175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:09.0913048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:09.4613325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:09.4613850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:09.4614243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:09.4614935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:09.4615569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:09.4616170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:09.4720084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:09.4720799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:09.5625564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:09.5834919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:39:09.5835501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:39:09.5836046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:39:09.5836725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:09.5837330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:09.5837885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:11.1686936Z ok (3.376s) 2022-08-17T13:39:11.1687403Z 2022-08-17T13:39:11.1688381Z ---------------------------------------------------------------------- 2022-08-17T13:39:11.1688793Z Ran 1 test in 3.376s 2022-08-17T13:39:11.1688910Z 2022-08-17T13:39:11.1688973Z OK 2022-08-17T13:39:11.1689077Z 2022-08-17T13:39:11.1689173Z Generating XML reports... 2022-08-17T13:39:11.1726257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133907.xml 2022-08-17T13:39:12.4726914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:12.4727613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:12.4730607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:12.4731202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:12.8220577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:12.8231047Z 2022-08-17T13:39:12.8231369Z Running tests... 2022-08-17T13:39:12.8232001Z ---------------------------------------------------------------------- 2022-08-17T13:39:13.0936502Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22506 2022-08-17T13:39:13.0966326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22507 2022-08-17T13:39:13.0994699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22508 2022-08-17T13:39:14.0791704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:14.0792111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:14.0792860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:14.0793298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:14.0794689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:14.0795112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:14.0795626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:14.0795978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:14.0929635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:14.0930001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:14.0933692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:14.0934053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:14.4613051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:14.4685036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:14.4685555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:14.4686533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:14.4687102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:14.4713827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:14.4794457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:14.5726066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:14.5797283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:14.9043437Z skip: CUDA is not available. (2.081s) 2022-08-17T13:39:14.9043965Z 2022-08-17T13:39:14.9044535Z ---------------------------------------------------------------------- 2022-08-17T13:39:14.9044805Z Ran 1 test in 2.081s 2022-08-17T13:39:14.9044926Z 2022-08-17T13:39:14.9045001Z OK (skipped=1) 2022-08-17T13:39:14.9045101Z 2022-08-17T13:39:14.9045191Z Generating XML reports... 2022-08-17T13:39:14.9079861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133912.xml 2022-08-17T13:39:16.2030738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:16.2031132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:16.2034544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:16.2034991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:16.5428833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:16.5438254Z 2022-08-17T13:39:16.5438569Z Running tests... 2022-08-17T13:39:16.5439195Z ---------------------------------------------------------------------- 2022-08-17T13:39:16.8132791Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22559 2022-08-17T13:39:16.8161753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22560 2022-08-17T13:39:16.8190457Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22561 2022-08-17T13:39:17.8128424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:17.8129104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:17.8131094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:17.8131684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:17.8186008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:17.8189936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:17.8190827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:17.8191490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:17.8400135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:17.8400780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:17.8403113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:17.8403789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:18.1922979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:18.1969930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:18.1970487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:18.1971208Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:18.1971795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:18.2023714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:18.2077451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:18.2079167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:18.2284270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:39:18.2284661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:39:18.3033919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:18.3036572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:39:18.3037489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:18.3093940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:18.3094560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:19.5251304Z ok (2.981s) 2022-08-17T13:39:19.5251519Z 2022-08-17T13:39:19.5252047Z ---------------------------------------------------------------------- 2022-08-17T13:39:19.5252452Z Ran 1 test in 2.981s 2022-08-17T13:39:19.5252574Z 2022-08-17T13:39:19.5252643Z OK 2022-08-17T13:39:19.5252738Z 2022-08-17T13:39:19.5252831Z Generating XML reports... 2022-08-17T13:39:19.5289081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133916.xml 2022-08-17T13:39:20.8577368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:20.8577785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:20.8580182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:20.8580560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:21.2091552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:21.2101321Z 2022-08-17T13:39:21.2101422Z Running tests... 2022-08-17T13:39:21.2101893Z ---------------------------------------------------------------------- 2022-08-17T13:39:21.4774489Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22618 2022-08-17T13:39:21.4801701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22619 2022-08-17T13:39:21.4830495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22620 2022-08-17T13:39:22.4727177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:22.4728127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:22.4729845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:22.4730498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:22.4774737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:22.4775391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:22.4779479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:22.4779922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:22.4863087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:22.4863551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:22.4867376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:22.4867891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:22.8653911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:22.8755634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:22.8756196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:22.8756840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:22.8757397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:22.8757945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:22.8764448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:22.8765070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:22.8766704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:23.1875572Z skip: CUDA is not available. (1.977s) 2022-08-17T13:39:23.1875900Z 2022-08-17T13:39:23.1876444Z ---------------------------------------------------------------------- 2022-08-17T13:39:23.1876939Z Ran 1 test in 1.977s 2022-08-17T13:39:23.1877064Z 2022-08-17T13:39:23.1877142Z OK (skipped=1) 2022-08-17T13:39:23.1877241Z 2022-08-17T13:39:23.1877332Z Generating XML reports... 2022-08-17T13:39:23.1912723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133921.xml 2022-08-17T13:39:24.4599040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:24.4599734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:24.4602915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:24.4603336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:24.7987256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:24.7997071Z 2022-08-17T13:39:24.7997207Z Running tests... 2022-08-17T13:39:24.7997819Z ---------------------------------------------------------------------- 2022-08-17T13:39:25.0636797Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22671 2022-08-17T13:39:25.0664103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22672 2022-08-17T13:39:25.0693876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22673 2022-08-17T13:39:26.0393677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:26.0394327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:26.0396249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:26.0396856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:26.0402585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:26.0403213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:26.0407109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:26.0407756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:26.0416377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:26.0417023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:26.0421511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:26.0422120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:26.4020113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:26.4120901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:26.4121559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:26.4122599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:26.4123450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:26.4124363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:26.4228927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:26.5134282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:26.5135190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:26.5343112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:39:26.5444233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:39:26.5444835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:39:26.5445628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:26.5446664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:26.5447498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:27.9758958Z ok (3.176s) 2022-08-17T13:39:27.9759224Z 2022-08-17T13:39:27.9759689Z ---------------------------------------------------------------------- 2022-08-17T13:39:27.9760004Z Ran 1 test in 3.176s 2022-08-17T13:39:27.9760131Z 2022-08-17T13:39:27.9760197Z OK 2022-08-17T13:39:27.9760298Z 2022-08-17T13:39:27.9760392Z Generating XML reports... 2022-08-17T13:39:27.9798206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133924.xml 2022-08-17T13:39:29.2547787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:29.2548274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:29.2550134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:29.2550513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:29.5998049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:29.6007396Z 2022-08-17T13:39:29.6007545Z Running tests... 2022-08-17T13:39:29.6008173Z ---------------------------------------------------------------------- 2022-08-17T13:39:29.8627298Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22733 2022-08-17T13:39:29.8654631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22734 2022-08-17T13:39:29.8682049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22735 2022-08-17T13:39:30.8367751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:30.8368355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:30.8370947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:30.8371634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:30.8891801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:30.8892427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:30.8895433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:30.8895847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:30.9113797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:30.9114163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:30.9117221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:30.9117605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:31.2791092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:31.2791766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:31.2792704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:31.2793408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:31.2794165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:31.2794720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:31.2896944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:31.3803893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:31.3804289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:31.4323816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:31.4424504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:31.4425263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:31.4426142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:31.4426705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:31.4427245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:32.8749333Z ok (3.274s) 2022-08-17T13:39:32.8749551Z 2022-08-17T13:39:32.8749883Z ---------------------------------------------------------------------- 2022-08-17T13:39:32.8750153Z Ran 1 test in 3.274s 2022-08-17T13:39:32.8750274Z 2022-08-17T13:39:32.8750339Z OK 2022-08-17T13:39:32.8750436Z 2022-08-17T13:39:32.8750550Z Generating XML reports... 2022-08-17T13:39:32.8785506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133929.xml 2022-08-17T13:39:34.1687778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:34.1688167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:34.1690341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:34.1690795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:34.5149772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:34.5159443Z 2022-08-17T13:39:34.5159604Z Running tests... 2022-08-17T13:39:34.5160469Z ---------------------------------------------------------------------- 2022-08-17T13:39:34.7828460Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22795 2022-08-17T13:39:34.7856892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22796 2022-08-17T13:39:34.7885185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22797 2022-08-17T13:39:35.7801722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:35.7802318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:35.7804968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:35.7806067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:35.8095467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:35.8096145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:35.8097550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:35.8098220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:35.8148534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:35.8148924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:35.8153095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:35.8153782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:36.1822929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:36.1823738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:36.1824139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:36.1824787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:36.1825352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:36.1825886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:36.1930635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:36.1931288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:36.2135856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:39:36.2136256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:39:36.2834242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:36.2836445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:39:36.2837122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:36.2845353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:36.2846259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:39:41.6001448Z ok (7.084s) 2022-08-17T13:39:41.6001702Z 2022-08-17T13:39:41.6002053Z ---------------------------------------------------------------------- 2022-08-17T13:39:41.6002306Z Ran 1 test in 7.084s 2022-08-17T13:39:41.6002519Z 2022-08-17T13:39:41.6002588Z OK 2022-08-17T13:39:41.6002685Z 2022-08-17T13:39:41.6002780Z Generating XML reports... 2022-08-17T13:39:41.6038140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133934.xml 2022-08-17T13:39:42.9118415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:42.9119112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:42.9121934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:42.9122812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:43.2559230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:43.2569892Z 2022-08-17T13:39:43.2570225Z Running tests... 2022-08-17T13:39:43.2570917Z ---------------------------------------------------------------------- 2022-08-17T13:39:43.5185165Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22854 2022-08-17T13:39:43.5212527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22855 2022-08-17T13:39:43.5239939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22856 2022-08-17T13:39:44.5444157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:44.5444561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:44.5447524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:44.5447999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:44.5498207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:44.5498572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:44.5502531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:44.5503190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:44.5657190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:44.5657891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:44.5661187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:44.5661814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:44.9264604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:44.9366053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:44.9366548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:44.9367214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:44.9367759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:44.9368301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:44.9473151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:45.0378775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:45.0379385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:45.3292497Z ok (2.072s) 2022-08-17T13:39:45.3292762Z 2022-08-17T13:39:45.3293279Z ---------------------------------------------------------------------- 2022-08-17T13:39:45.3293697Z Ran 1 test in 2.072s 2022-08-17T13:39:45.3293819Z 2022-08-17T13:39:45.3293870Z OK 2022-08-17T13:39:45.3293968Z 2022-08-17T13:39:45.3294066Z Generating XML reports... 2022-08-17T13:39:45.3331271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133943.xml 2022-08-17T13:39:46.6238925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:46.6239559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:46.6241830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:46.6242360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:46.9727457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:46.9737123Z 2022-08-17T13:39:46.9737287Z Running tests... 2022-08-17T13:39:46.9737778Z ---------------------------------------------------------------------- 2022-08-17T13:39:47.2422746Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22907 2022-08-17T13:39:47.2452794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22908 2022-08-17T13:39:47.2482656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22909 2022-08-17T13:39:48.2299756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:48.2300182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:48.2302687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:48.2303204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:48.2339613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:48.2340236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:48.2344453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:48.2345041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:48.2534515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:48.2534902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:48.2537875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:48.2538409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:48.6109885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:39:48.6210762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:39:48.6211496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:39:48.6212405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:48.6213170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:48.6213721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:39:48.6318477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:39:48.7226955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:39:48.7227353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:39:49.1535223Z ok (2.179s) 2022-08-17T13:39:49.1535442Z 2022-08-17T13:39:49.1535891Z ---------------------------------------------------------------------- 2022-08-17T13:39:49.1536269Z Ran 1 test in 2.180s 2022-08-17T13:39:49.1536452Z 2022-08-17T13:39:49.1536551Z OK 2022-08-17T13:39:49.1536689Z 2022-08-17T13:39:49.1537135Z Generating XML reports... 2022-08-17T13:39:49.1574332Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133946.xml 2022-08-17T13:39:50.4401386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:50.4402086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:50.4404707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:50.4405154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:50.7789590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:50.7798959Z 2022-08-17T13:39:50.7799066Z Running tests... 2022-08-17T13:39:50.7799981Z ---------------------------------------------------------------------- 2022-08-17T13:39:50.7824790Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:50.7825309Z 2022-08-17T13:39:50.7825641Z ---------------------------------------------------------------------- 2022-08-17T13:39:50.7825901Z Ran 1 test in 0.003s 2022-08-17T13:39:50.7826022Z 2022-08-17T13:39:50.7826085Z OK (skipped=1) 2022-08-17T13:39:50.7826435Z 2022-08-17T13:39:50.7826527Z Generating XML reports... 2022-08-17T13:39:50.7850077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133950.xml 2022-08-17T13:39:51.9455241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:51.9455834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:51.9457411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:51.9458050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:52.2895445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:52.2905880Z 2022-08-17T13:39:52.2906158Z Running tests... 2022-08-17T13:39:52.2906813Z ---------------------------------------------------------------------- 2022-08-17T13:39:52.2934455Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2022-08-17T13:39:52.2934852Z 2022-08-17T13:39:52.2935261Z ---------------------------------------------------------------------- 2022-08-17T13:39:52.2935660Z Ran 1 test in 0.003s 2022-08-17T13:39:52.2935854Z 2022-08-17T13:39:52.2935979Z OK (skipped=1) 2022-08-17T13:39:52.2936158Z 2022-08-17T13:39:52.2936309Z Generating XML reports... 2022-08-17T13:39:52.2961604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133952.xml 2022-08-17T13:39:53.4629526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:53.4630193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:53.4632707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:53.4633138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:53.8097785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:53.8107937Z 2022-08-17T13:39:53.8108036Z Running tests... 2022-08-17T13:39:53.8108652Z ---------------------------------------------------------------------- 2022-08-17T13:39:53.8133614Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:53.8133902Z 2022-08-17T13:39:53.8134160Z ---------------------------------------------------------------------- 2022-08-17T13:39:53.8134682Z Ran 1 test in 0.003s 2022-08-17T13:39:53.8134804Z 2022-08-17T13:39:53.8134880Z OK (skipped=1) 2022-08-17T13:39:53.8134994Z 2022-08-17T13:39:53.8135086Z Generating XML reports... 2022-08-17T13:39:53.8161117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133953.xml 2022-08-17T13:39:54.9758875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:54.9759287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:54.9762046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:54.9762415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:55.3287203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:55.3297021Z 2022-08-17T13:39:55.3297196Z Running tests... 2022-08-17T13:39:55.3297613Z ---------------------------------------------------------------------- 2022-08-17T13:39:55.3318005Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:55.3318317Z 2022-08-17T13:39:55.3318952Z ---------------------------------------------------------------------- 2022-08-17T13:39:55.3319311Z Ran 1 test in 0.002s 2022-08-17T13:39:55.3319433Z 2022-08-17T13:39:55.3319515Z OK (skipped=1) 2022-08-17T13:39:55.3319628Z 2022-08-17T13:39:55.3319721Z Generating XML reports... 2022-08-17T13:39:55.3344920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133955.xml 2022-08-17T13:39:56.4945803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:56.4946199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:56.4948362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:56.4948744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:56.8388157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:56.8397909Z 2022-08-17T13:39:56.8398217Z Running tests... 2022-08-17T13:39:56.8398569Z ---------------------------------------------------------------------- 2022-08-17T13:39:56.8417589Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:56.8417896Z 2022-08-17T13:39:56.8418226Z ---------------------------------------------------------------------- 2022-08-17T13:39:56.8418485Z Ran 1 test in 0.002s 2022-08-17T13:39:56.8418592Z 2022-08-17T13:39:56.8418674Z OK (skipped=1) 2022-08-17T13:39:56.8418847Z 2022-08-17T13:39:56.8418968Z Generating XML reports... 2022-08-17T13:39:56.8444644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133956.xml 2022-08-17T13:39:57.9990671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:57.9991082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:57.9993762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:57.9994138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:58.3479901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:58.3489890Z 2022-08-17T13:39:58.3489998Z Running tests... 2022-08-17T13:39:58.3491465Z ---------------------------------------------------------------------- 2022-08-17T13:39:58.3514294Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:58.3514847Z 2022-08-17T13:39:58.3515113Z ---------------------------------------------------------------------- 2022-08-17T13:39:58.3515378Z Ran 1 test in 0.002s 2022-08-17T13:39:58.3515499Z 2022-08-17T13:39:58.3515578Z OK (skipped=1) 2022-08-17T13:39:58.3515693Z 2022-08-17T13:39:58.3515777Z Generating XML reports... 2022-08-17T13:39:58.3540372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133958.xml 2022-08-17T13:39:59.5060157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:39:59.5060569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:39:59.5063466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:39:59.5063867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:39:59.8459735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:39:59.8469467Z 2022-08-17T13:39:59.8469570Z Running tests... 2022-08-17T13:39:59.8470007Z ---------------------------------------------------------------------- 2022-08-17T13:39:59.8493319Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:39:59.8493679Z 2022-08-17T13:39:59.8494080Z ---------------------------------------------------------------------- 2022-08-17T13:39:59.8494463Z Ran 1 test in 0.002s 2022-08-17T13:39:59.8494582Z 2022-08-17T13:39:59.8494659Z OK (skipped=1) 2022-08-17T13:39:59.8494772Z 2022-08-17T13:39:59.8494860Z Generating XML reports... 2022-08-17T13:39:59.8520710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133959.xml 2022-08-17T13:40:01.0025163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:01.0025860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:01.0028518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:01.0029141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:01.3463477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:01.3473206Z 2022-08-17T13:40:01.3473498Z Running tests... 2022-08-17T13:40:01.3474119Z ---------------------------------------------------------------------- 2022-08-17T13:40:01.3494267Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-08-17T13:40:01.3494744Z 2022-08-17T13:40:01.3495125Z ---------------------------------------------------------------------- 2022-08-17T13:40:01.3495373Z Ran 1 test in 0.002s 2022-08-17T13:40:01.3495510Z 2022-08-17T13:40:01.3495586Z OK (skipped=1) 2022-08-17T13:40:01.3495699Z 2022-08-17T13:40:01.3495789Z Generating XML reports... 2022-08-17T13:40:01.3521761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134001.xml 2022-08-17T13:40:02.5346261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:02.5346646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:02.5349488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:02.5349866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:02.8836435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:02.8846535Z 2022-08-17T13:40:02.8846832Z Running tests... 2022-08-17T13:40:02.8847431Z ---------------------------------------------------------------------- 2022-08-17T13:40:03.1537197Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23040 2022-08-17T13:40:03.1567974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23041 2022-08-17T13:40:03.1597906Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23042 2022-08-17T13:40:04.1307276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:04.1307729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:04.1309758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:04.1310139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:04.1416487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:04.1417166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:04.1420506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:04.1421257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:04.1674397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:04.1674775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:04.1678111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:04.1678489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:04.5251724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:04.5352258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:04.5352909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:04.5353568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:04.5354110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:04.5354661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:04.5461041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:04.5461784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:04.5464474Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.5465686Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6366635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:04.6368922Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6374923Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6375687Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6376448Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6385124Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6390893Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6391737Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6396158Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6396983Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6397694Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6401564Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6405427Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6407107Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6409305Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6410761Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6411325Z 2022-08-17T13:40:04.6415527Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6417004Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6418421Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6422408Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6423580Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6424148Z 2022-08-17T13:40:04.6426997Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6429236Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6430779Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6434391Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6434880Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6435351Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6438925Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6441530Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6442379Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6444638Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6445435Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6446393Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6449415Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6450777Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6454032Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6454869Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6457587Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6458434Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6461655Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6464112Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6464945Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6466790Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6468011Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6468601Z 2022-08-17T13:40:04.6470880Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6474494Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6475312Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6477341Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6478135Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6478982Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6481401Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6484080Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6485718Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6487024Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6489613Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6490131Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6495298Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6496018Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6496541Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6499362Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6500142Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6500940Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6504608Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6506939Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6509070Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6512538Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6513354Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6514372Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6520291Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6520943Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6521549Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6522244Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6533269Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6533849Z 2022-08-17T13:40:04.6534330Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6535108Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6535819Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6537030Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6537844Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6538656Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6543708Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6544404Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6545144Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6548162Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6548862Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6549342Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6552754Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6554578Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6555051Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6562698Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6564315Z 2022-08-17T13:40:04.6564881Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6565572Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6566378Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6566950Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6573822Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6574320Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6574785Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6575251Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6576011Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6576806Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:04.6579362Z STAGE:2022-08-17 13:40:04 23040:23040 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6579938Z STAGE:2022-08-17 13:40:04 23042:23042 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:04.6580463Z STAGE:2022-08-17 13:40:04 23041:23041 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:05.0647950Z ok (2.180s) 2022-08-17T13:40:05.0648175Z 2022-08-17T13:40:05.0648666Z ---------------------------------------------------------------------- 2022-08-17T13:40:05.0649066Z Ran 1 test in 2.180s 2022-08-17T13:40:05.0649254Z 2022-08-17T13:40:05.0649357Z OK 2022-08-17T13:40:05.0649521Z 2022-08-17T13:40:05.0649653Z Generating XML reports... 2022-08-17T13:40:05.0685432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134002.xml 2022-08-17T13:40:06.3575151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:06.3575550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:06.3577891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:06.3578285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:06.6970943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:06.6980134Z 2022-08-17T13:40:06.6980227Z Running tests... 2022-08-17T13:40:06.6981279Z ---------------------------------------------------------------------- 2022-08-17T13:40:06.9539605Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s) 2022-08-17T13:40:06.9540500Z 2022-08-17T13:40:06.9540876Z ---------------------------------------------------------------------- 2022-08-17T13:40:06.9541336Z Ran 1 test in 0.256s 2022-08-17T13:40:06.9541538Z 2022-08-17T13:40:06.9541666Z OK (skipped=1) 2022-08-17T13:40:06.9541852Z 2022-08-17T13:40:06.9542001Z Generating XML reports... 2022-08-17T13:40:06.9564880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134006.xml 2022-08-17T13:40:08.1893795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:08.1894440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:08.1896462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:08.1897097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:08.5317743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:08.5328107Z 2022-08-17T13:40:08.5328607Z Running tests... 2022-08-17T13:40:08.5329028Z ---------------------------------------------------------------------- 2022-08-17T13:40:08.8003026Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23106 2022-08-17T13:40:08.8032327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23107 2022-08-17T13:40:08.8061273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23108 2022-08-17T13:40:09.8286143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:09.8286817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:09.8290052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:09.8290766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:09.8540064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:09.8540723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:09.8544363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:09.8544793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:09.8917288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:09.8917723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:09.8919874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:09.8920261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:10.2474614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:10.2575816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:10.2576443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:10.2577186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:10.2578135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:10.2578929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:10.2682554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:10.3588269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:10.3588857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:10.3796235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:40:10.3898613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:40:10.3899236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:40:10.3899977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:10.3900861Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:10.3901619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:10.3902533Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3903597Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3903977Z 2022-08-17T13:40:10.3909580Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3910722Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3911416Z 2022-08-17T13:40:10.3918370Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3927985Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3928619Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3931966Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3939058Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3943746Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3944532Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3948894Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3953316Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3956016Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3959827Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3960739Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3962268Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3965279Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3968534Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3971897Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3972906Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3975382Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3975966Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3982301Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3985042Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3988503Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3989904Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.3990428Z 2022-08-17T13:40:10.3994628Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3995378Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.3996132Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4000212Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4000893Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4001355Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4009066Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4010107Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4010660Z 2022-08-17T13:40:10.4013666Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4014273Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4015903Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4022547Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4023198Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4024037Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4027862Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4028185Z 2022-08-17T13:40:10.4028458Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4032474Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4034514Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4037056Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4037662Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4041437Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4041757Z 2022-08-17T13:40:10.4048508Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4050207Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4050871Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4051510Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4052049Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4056021Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4058616Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4059265Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4065441Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4069470Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4070291Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4071108Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4075217Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4075946Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4077443Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4081568Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4082438Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4083253Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4086093Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4089312Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4089864Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4100024Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4100891Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4101363Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4101962Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4102779Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4103135Z 2022-08-17T13:40:10.4104047Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4104766Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4112047Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4112841Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4113647Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4113943Z 2022-08-17T13:40:10.4117832Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4118163Z 2022-08-17T13:40:10.4118441Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4123939Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4124703Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4125439Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4130188Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4130762Z 2022-08-17T13:40:10.4131260Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4135061Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4135691Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4136877Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4144960Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4145969Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4146775Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4147623Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4148387Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4149109Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:10.4150639Z STAGE:2022-08-17 13:40:10 23107:23107 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4153793Z STAGE:2022-08-17 13:40:10 23108:23108 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.4154618Z STAGE:2022-08-17 13:40:10 23106:23106 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:10.8112357Z ok (2.278s) 2022-08-17T13:40:10.8112656Z 2022-08-17T13:40:10.8113048Z ---------------------------------------------------------------------- 2022-08-17T13:40:10.8113459Z Ran 1 test in 2.278s 2022-08-17T13:40:10.8113968Z 2022-08-17T13:40:10.8114065Z OK 2022-08-17T13:40:10.8114152Z 2022-08-17T13:40:10.8114249Z Generating XML reports... 2022-08-17T13:40:10.8149110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134008.xml 2022-08-17T13:40:12.0885284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:12.0886037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:12.0887310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:12.0887947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:12.4340173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:12.4349942Z 2022-08-17T13:40:12.4350085Z Running tests... 2022-08-17T13:40:12.4350710Z ---------------------------------------------------------------------- 2022-08-17T13:40:12.7016860Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23171 2022-08-17T13:40:12.7044989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23172 2022-08-17T13:40:12.7073134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23173 2022-08-17T13:40:13.6984589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:13.6985220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:13.6985982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:13.6986591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:13.6987643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:13.6988238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:13.6989007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:13.6989629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:13.7412134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:13.7412866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:13.7415331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:13.7416337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:14.1017585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:14.1118792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:14.1119942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:14.1120598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:14.1121110Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:14.1121654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:14.1129671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:14.1130366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:14.1131230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:14.1131763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:40:14.1338723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:40:14.1339358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:40:14.1340377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:14.1341215Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:14.1342675Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1343403Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1347702Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1348167Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1355701Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1361048Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1365216Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1365540Z 2022-08-17T13:40:14.1370001Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1375003Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1378143Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1378974Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1382672Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1386630Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1389440Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1393846Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1396162Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1402626Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1405488Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1406170Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1409753Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1413119Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1415651Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1416147Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1420165Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1422687Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1425242Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1427486Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1428956Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1435286Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1436203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:40:14.1438099Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1440011Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1443887Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1446747Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1449205Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1449727Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1453474Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1455288Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1457964Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1458765Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1462173Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1464986Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1468009Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1469131Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1474070Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1474555Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:40:14.1477286Z STAGE:2022-08-17 13:40:14 23173:23173 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.1477822Z STAGE:2022-08-17 13:40:14 23172:23172 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:40:14.6124640Z ok (2.177s) 2022-08-17T13:40:14.6124961Z 2022-08-17T13:40:14.6125302Z ---------------------------------------------------------------------- 2022-08-17T13:40:14.6125561Z Ran 1 test in 2.177s 2022-08-17T13:40:14.6125688Z 2022-08-17T13:40:14.6125741Z OK 2022-08-17T13:40:14.6125998Z 2022-08-17T13:40:14.6126095Z Generating XML reports... 2022-08-17T13:40:14.6160518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134012.xml 2022-08-17T13:40:15.9070101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:15.9070483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:15.9072856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:15.9073232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:16.2520812Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:16.2531097Z 2022-08-17T13:40:16.2531414Z Running tests... 2022-08-17T13:40:16.2532003Z ---------------------------------------------------------------------- 2022-08-17T13:40:16.5235151Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23232 2022-08-17T13:40:16.5263886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23233 2022-08-17T13:40:16.5291935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23234 2022-08-17T13:40:17.5342687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:17.5357098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:17.5358147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:17.5358801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:17.5359593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:17.5360167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:17.5361010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:17.5361648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:17.5420026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:17.5420652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:17.5424844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:17.5425468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:17.9029433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:17.9129881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:17.9130636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:17.9131292Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:17.9131833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:17.9132582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:17.9236509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:18.0142534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:18.0143134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:18.3341570Z skip: CUDA is not available. (2.081s) 2022-08-17T13:40:18.3341900Z 2022-08-17T13:40:18.3342493Z ---------------------------------------------------------------------- 2022-08-17T13:40:18.3342980Z Ran 1 test in 2.081s 2022-08-17T13:40:18.3343507Z 2022-08-17T13:40:18.3343745Z OK (skipped=1) 2022-08-17T13:40:18.3343954Z 2022-08-17T13:40:18.3344125Z Generating XML reports... 2022-08-17T13:40:18.3377769Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134016.xml 2022-08-17T13:40:19.6311721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:19.6312111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:19.6315006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:19.6315387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:19.9893767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:19.9904797Z 2022-08-17T13:40:19.9905063Z Running tests... 2022-08-17T13:40:19.9905690Z ---------------------------------------------------------------------- 2022-08-17T13:40:20.2490220Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s) 2022-08-17T13:40:20.2491294Z 2022-08-17T13:40:20.2491682Z ---------------------------------------------------------------------- 2022-08-17T13:40:20.2492086Z Ran 1 test in 0.258s 2022-08-17T13:40:20.2492284Z 2022-08-17T13:40:20.2492428Z OK (skipped=1) 2022-08-17T13:40:20.2492617Z 2022-08-17T13:40:20.2492765Z Generating XML reports... 2022-08-17T13:40:20.2516658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134019.xml 2022-08-17T13:40:21.4921291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:21.4921670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:21.4924252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:21.4924776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:21.8390120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:21.8399754Z 2022-08-17T13:40:21.8400104Z Running tests... 2022-08-17T13:40:21.8400527Z ---------------------------------------------------------------------- 2022-08-17T13:40:21.8419102Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:40:21.8419659Z 2022-08-17T13:40:21.8420047Z ---------------------------------------------------------------------- 2022-08-17T13:40:21.8420311Z Ran 1 test in 0.002s 2022-08-17T13:40:21.8420431Z 2022-08-17T13:40:21.8420507Z OK (skipped=1) 2022-08-17T13:40:21.8420619Z 2022-08-17T13:40:21.8420707Z Generating XML reports... 2022-08-17T13:40:21.8444799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134021.xml 2022-08-17T13:40:23.0108935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:23.0109343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:23.0111028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:23.0111444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:23.3520645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:23.3531488Z 2022-08-17T13:40:23.3531901Z Running tests... 2022-08-17T13:40:23.3532285Z ---------------------------------------------------------------------- 2022-08-17T13:40:23.3549327Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:40:23.3549925Z 2022-08-17T13:40:23.3550325Z ---------------------------------------------------------------------- 2022-08-17T13:40:23.3550649Z Ran 1 test in 0.002s 2022-08-17T13:40:23.3550771Z 2022-08-17T13:40:23.3551072Z OK (skipped=1) 2022-08-17T13:40:23.3551176Z 2022-08-17T13:40:23.3551265Z Generating XML reports... 2022-08-17T13:40:23.3576952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134023.xml 2022-08-17T13:40:24.5103795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:24.5104190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:24.5106986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:24.5107507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:24.8632588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:24.8642387Z 2022-08-17T13:40:24.8642642Z Running tests... 2022-08-17T13:40:24.8643279Z ---------------------------------------------------------------------- 2022-08-17T13:40:25.1322004Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23315 2022-08-17T13:40:25.1351703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23316 2022-08-17T13:40:25.1380750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23317 2022-08-17T13:40:26.1261310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:26.1261698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:26.1264868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:26.1265358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:26.1424200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:26.1424664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:26.1427709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:26.1428228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:26.1503060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:26.1503535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:26.1507285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:26.1507969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:26.5254984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:26.5255776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:26.5256178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:26.5256994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:26.5257876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:26.5258414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:26.5263283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:26.5265447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:26.5266255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:26.8430739Z skip: Need at least 2 CUDA devices (1.978s) 2022-08-17T13:40:26.8431102Z 2022-08-17T13:40:26.8431622Z ---------------------------------------------------------------------- 2022-08-17T13:40:26.8431999Z Ran 1 test in 1.979s 2022-08-17T13:40:26.8432120Z 2022-08-17T13:40:26.8432197Z OK (skipped=1) 2022-08-17T13:40:26.8432312Z 2022-08-17T13:40:26.8432388Z Generating XML reports... 2022-08-17T13:40:26.8468341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134024.xml 2022-08-17T13:40:28.1523463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:28.1523868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:28.1526491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:28.1526881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:28.4963424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:28.4973223Z 2022-08-17T13:40:28.4973322Z Running tests... 2022-08-17T13:40:28.4974235Z ---------------------------------------------------------------------- 2022-08-17T13:40:28.7668306Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23368 2022-08-17T13:40:28.7698291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23369 2022-08-17T13:40:28.7727053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23370 2022-08-17T13:40:29.7579264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:29.7579659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:29.7583332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:29.7583996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:29.7624894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:29.7625487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:29.7629738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:29.7630276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:29.8050564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:29.8050944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:29.8053488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:29.8053863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:30.1644898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:30.1746006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:30.1746673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:30.1747655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:30.1748620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:30.1749533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:30.1853231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:30.2759765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:30.2760381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:30.5775908Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T13:40:30.5776250Z 2022-08-17T13:40:30.5776750Z ---------------------------------------------------------------------- 2022-08-17T13:40:30.5777218Z Ran 1 test in 2.080s 2022-08-17T13:40:30.5777411Z 2022-08-17T13:40:30.5777578Z OK (skipped=1) 2022-08-17T13:40:30.5777780Z 2022-08-17T13:40:30.5777941Z Generating XML reports... 2022-08-17T13:40:30.5815206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134028.xml 2022-08-17T13:40:31.8886696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:31.8887114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:31.8888856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:31.8889318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:32.2367725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:32.2377332Z 2022-08-17T13:40:32.2377490Z Running tests... 2022-08-17T13:40:32.2377965Z ---------------------------------------------------------------------- 2022-08-17T13:40:32.5012173Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23421 2022-08-17T13:40:32.5040315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23422 2022-08-17T13:40:32.5067923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23423 2022-08-17T13:40:33.4837955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:33.4838609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:33.4841085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:33.4841733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:33.4919349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:33.4920219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:33.4923098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:33.4923753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:33.4963507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:33.4964114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:33.4968830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:33.4969449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:33.8588286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:33.8588967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:33.8589739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:33.8590545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:33.8591111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:33.8591660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:33.8694709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:33.9602138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:33.9602597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:34.3115252Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:40:34.3115573Z 2022-08-17T13:40:34.3116099Z ---------------------------------------------------------------------- 2022-08-17T13:40:34.3116361Z Ran 1 test in 2.074s 2022-08-17T13:40:34.3116482Z 2022-08-17T13:40:34.3116558Z OK (skipped=1) 2022-08-17T13:40:34.3116685Z 2022-08-17T13:40:34.3116775Z Generating XML reports... 2022-08-17T13:40:34.3151992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134032.xml 2022-08-17T13:40:35.6093260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:35.6093662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:35.6095908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:35.6096480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:35.9485049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:35.9495186Z 2022-08-17T13:40:35.9495329Z Running tests... 2022-08-17T13:40:35.9495795Z ---------------------------------------------------------------------- 2022-08-17T13:40:36.2146873Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23474 2022-08-17T13:40:36.2173629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23475 2022-08-17T13:40:36.2200873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23476 2022-08-17T13:40:37.1972196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:37.1972866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:37.1975318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:37.1975781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:37.2169936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:37.2170307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:37.2173186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:37.2173612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:37.2543687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:37.2544042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:37.2547632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:37.2548231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:37.6174770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:37.6276206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:37.6276711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:37.6277617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:37.6278533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:37.6279125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:37.6284168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:37.6284619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:37.6289031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:37.9251036Z skip: Need at least 2 CUDA devices (1.975s) 2022-08-17T13:40:37.9251280Z 2022-08-17T13:40:37.9251636Z ---------------------------------------------------------------------- 2022-08-17T13:40:37.9251896Z Ran 1 test in 1.975s 2022-08-17T13:40:37.9252016Z 2022-08-17T13:40:37.9252151Z OK (skipped=1) 2022-08-17T13:40:37.9252270Z 2022-08-17T13:40:37.9252345Z Generating XML reports... 2022-08-17T13:40:37.9286029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134035.xml 2022-08-17T13:40:39.2066270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:39.2066666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:39.2068510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:39.2068896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:39.5505929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:39.5515825Z 2022-08-17T13:40:39.5515906Z Running tests... 2022-08-17T13:40:39.5516494Z ---------------------------------------------------------------------- 2022-08-17T13:40:39.5548554Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:40:39.5549066Z 2022-08-17T13:40:39.5549423Z ---------------------------------------------------------------------- 2022-08-17T13:40:39.5549913Z Ran 1 test in 0.003s 2022-08-17T13:40:39.5550034Z 2022-08-17T13:40:39.5550110Z OK (skipped=1) 2022-08-17T13:40:39.5550224Z 2022-08-17T13:40:39.5550312Z Generating XML reports... 2022-08-17T13:40:39.5573815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134039.xml 2022-08-17T13:40:40.7073211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:40.7073858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:40.7080027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:40.7080589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:41.0491091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:41.0500264Z 2022-08-17T13:40:41.0500784Z Running tests... 2022-08-17T13:40:41.0501182Z ---------------------------------------------------------------------- 2022-08-17T13:40:41.3176652Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23537 2022-08-17T13:40:41.3206562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23538 2022-08-17T13:40:41.3236045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23539 2022-08-17T13:40:42.2811080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:42.2811740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:42.2813933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:42.2814574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:42.3161565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:42.3162217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:42.3164448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:42.3165089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:42.3189154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:42.3193700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:42.3194581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:42.3195241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:42.6990131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:42.7092308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:42.7092810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:42.7093466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:42.7094024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:42.7094559Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:42.7100362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:42.7100996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:42.7101744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:43.0282272Z skip: Need at least 2 CUDA devices (1.978s) 2022-08-17T13:40:43.0282552Z 2022-08-17T13:40:43.0282883Z ---------------------------------------------------------------------- 2022-08-17T13:40:43.0283163Z Ran 1 test in 1.978s 2022-08-17T13:40:43.0283358Z 2022-08-17T13:40:43.0283422Z OK (skipped=1) 2022-08-17T13:40:43.0283535Z 2022-08-17T13:40:43.0283626Z Generating XML reports... 2022-08-17T13:40:43.0319385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134041.xml 2022-08-17T13:40:44.3194990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:44.3195401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:44.3197897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:44.3198293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:44.6686669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:44.6696694Z 2022-08-17T13:40:44.6696799Z Running tests... 2022-08-17T13:40:44.6697682Z ---------------------------------------------------------------------- 2022-08-17T13:40:44.9378883Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23590 2022-08-17T13:40:44.9407737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23591 2022-08-17T13:40:44.9436180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23592 2022-08-17T13:40:45.9174105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:45.9174754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:45.9177070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:45.9177718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:45.9181427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:45.9182011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:45.9186603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:45.9187235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:45.9567532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:45.9568202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:45.9571832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:45.9572470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:46.3320695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:46.3422134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:46.3422791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:46.3423523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:46.3424424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:46.3425212Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:46.3529397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:46.4437709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:46.4438223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:46.7487338Z skip: Need at least 3 CUDA devices (2.079s) 2022-08-17T13:40:46.7487660Z 2022-08-17T13:40:46.7488203Z ---------------------------------------------------------------------- 2022-08-17T13:40:46.7488503Z Ran 1 test in 2.079s 2022-08-17T13:40:46.7488623Z 2022-08-17T13:40:46.7488699Z OK (skipped=1) 2022-08-17T13:40:46.7488801Z 2022-08-17T13:40:46.7488891Z Generating XML reports... 2022-08-17T13:40:46.7523518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134044.xml 2022-08-17T13:40:48.0548636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:48.0549036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:48.0550973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:48.0551337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:48.4007600Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:48.4017115Z 2022-08-17T13:40:48.4017267Z Running tests... 2022-08-17T13:40:48.4017667Z ---------------------------------------------------------------------- 2022-08-17T13:40:48.4060295Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s) 2022-08-17T13:40:48.4060780Z 2022-08-17T13:40:48.4061142Z ---------------------------------------------------------------------- 2022-08-17T13:40:48.4061558Z Ran 1 test in 0.004s 2022-08-17T13:40:48.4061760Z 2022-08-17T13:40:48.4061894Z OK (skipped=1) 2022-08-17T13:40:48.4062093Z 2022-08-17T13:40:48.4062335Z Generating XML reports... 2022-08-17T13:40:48.4087684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134048.xml 2022-08-17T13:40:49.5697072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:49.5697473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:49.5700383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:49.5700757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:49.9118474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:49.9128226Z 2022-08-17T13:40:49.9128369Z Running tests... 2022-08-17T13:40:49.9128994Z ---------------------------------------------------------------------- 2022-08-17T13:40:49.9165308Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s) 2022-08-17T13:40:49.9165819Z 2022-08-17T13:40:49.9166456Z ---------------------------------------------------------------------- 2022-08-17T13:40:49.9166924Z Ran 1 test in 0.004s 2022-08-17T13:40:49.9167141Z 2022-08-17T13:40:49.9167257Z OK (skipped=1) 2022-08-17T13:40:49.9167444Z 2022-08-17T13:40:49.9167554Z Generating XML reports... 2022-08-17T13:40:49.9191700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134049.xml 2022-08-17T13:40:51.0829945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:51.0830581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:51.0833112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:51.0833522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:51.4293552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:51.4303714Z 2022-08-17T13:40:51.4303803Z Running tests... 2022-08-17T13:40:51.4304310Z ---------------------------------------------------------------------- 2022-08-17T13:40:51.7032684Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23663 2022-08-17T13:40:51.7062246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23664 2022-08-17T13:40:51.7092314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23665 2022-08-17T13:40:52.6939536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:52.6940171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:52.6943048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:52.6943497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:52.7059096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:52.7059488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:52.7062675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:52.7063178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:52.7200129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:52.7200509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:52.7204434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:52.7204827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:53.0805168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:53.0852338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:53.0852823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:53.0853455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:53.0854013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:53.0905878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:53.0960439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:53.0960861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:40:53.1025053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93piilkf 2022-08-17T13:40:53.1025733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp719_h82u 2022-08-17T13:40:53.1027197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93piilkf/_remote_module_non_scriptable.py 2022-08-17T13:40:53.1028402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp719_h82u/_remote_module_non_scriptable.py 2022-08-17T13:40:53.1917629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:53.1983729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__5tcy_4 2022-08-17T13:40:53.1985870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__5tcy_4/_remote_module_non_scriptable.py 2022-08-17T13:40:53.2097933Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2099566Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2101026Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2103647Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:40:53.2104795Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:40:53.2106092Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:40:53.2107024Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:40:53.2108612Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:193: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.) 2022-08-17T13:40:53.2109963Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-08-17T13:40:53.2110501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:40:53.2111054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:40:53.2111707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:40:53.2113223Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2114566Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2115885Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2118007Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2119220Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2120405Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2123933Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2125704Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2127903Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2130153Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2131352Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2137962Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2142566Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2143894Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.2145913Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-08-17T13:40:53.5145213Z ok (2.084s) 2022-08-17T13:40:53.5145473Z 2022-08-17T13:40:53.5146010Z ---------------------------------------------------------------------- 2022-08-17T13:40:53.5146300Z Ran 1 test in 2.084s 2022-08-17T13:40:53.5146408Z 2022-08-17T13:40:53.5146474Z OK 2022-08-17T13:40:53.5146572Z 2022-08-17T13:40:53.5146672Z Generating XML reports... 2022-08-17T13:40:53.5181366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134051.xml 2022-08-17T13:40:54.8130119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:54.8130555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:54.8133527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:54.8133952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:55.1532043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:55.1541307Z 2022-08-17T13:40:55.1541658Z Running tests... 2022-08-17T13:40:55.1542110Z ---------------------------------------------------------------------- 2022-08-17T13:40:55.1586486Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s) 2022-08-17T13:40:55.1586811Z 2022-08-17T13:40:55.1587114Z ---------------------------------------------------------------------- 2022-08-17T13:40:55.1587430Z Ran 1 test in 0.005s 2022-08-17T13:40:55.1587582Z 2022-08-17T13:40:55.1587663Z OK (skipped=1) 2022-08-17T13:40:55.1587778Z 2022-08-17T13:40:55.1587879Z Generating XML reports... 2022-08-17T13:40:55.1612967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134055.xml 2022-08-17T13:40:56.2983179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:56.2983562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:56.2986486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:56.2987002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:56.6386481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:56.6395814Z 2022-08-17T13:40:56.6395964Z Running tests... 2022-08-17T13:40:56.6396596Z ---------------------------------------------------------------------- 2022-08-17T13:40:56.6426570Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:40:56.6427020Z 2022-08-17T13:40:56.6427253Z ---------------------------------------------------------------------- 2022-08-17T13:40:56.6427516Z Ran 1 test in 0.003s 2022-08-17T13:40:56.6427638Z 2022-08-17T13:40:56.6427734Z OK (skipped=1) 2022-08-17T13:40:56.6427907Z 2022-08-17T13:40:56.6427996Z Generating XML reports... 2022-08-17T13:40:56.6451648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134056.xml 2022-08-17T13:40:57.7965319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:57.7966293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:57.7967905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:57.7968885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:58.1466299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:40:58.1476456Z 2022-08-17T13:40:58.1476955Z Running tests... 2022-08-17T13:40:58.1477380Z ---------------------------------------------------------------------- 2022-08-17T13:40:58.4143434Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23736 2022-08-17T13:40:58.4171345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23737 2022-08-17T13:40:58.4199375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23738 2022-08-17T13:40:59.4067360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:59.4067762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:59.4070982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:59.4071673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:59.4283282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:59.4283913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:59.4286827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:59.4287243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:59.4294115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:40:59.4294603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:40:59.4298491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:40:59.4299061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:40:59.7980629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:40:59.8082536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:40:59.8083030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:40:59.8083751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:59.8084584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:59.8085146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:40:59.8091180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:40:59.8091923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:40:59.8093330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:00.1247189Z skip: Need at least 2 CUDA devices (1.977s) 2022-08-17T13:41:00.1247438Z 2022-08-17T13:41:00.1247767Z ---------------------------------------------------------------------- 2022-08-17T13:41:00.1248013Z Ran 1 test in 1.977s 2022-08-17T13:41:00.1248151Z 2022-08-17T13:41:00.1248226Z OK (skipped=1) 2022-08-17T13:41:00.1248339Z 2022-08-17T13:41:00.1248430Z Generating XML reports... 2022-08-17T13:41:00.1285407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134058.xml 2022-08-17T13:41:01.4035781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:01.4036357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:01.4037774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:01.4038136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:01.7453771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:01.7463496Z 2022-08-17T13:41:01.7463873Z Running tests... 2022-08-17T13:41:01.7464405Z ---------------------------------------------------------------------- 2022-08-17T13:41:02.0145032Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23789 2022-08-17T13:41:02.0172675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23790 2022-08-17T13:41:02.0201219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23791 2022-08-17T13:41:03.0101615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:03.0102227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:03.0105183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:03.0105851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:03.0137334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:03.0137981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:03.0142212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:03.0142892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:03.0336629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:03.0337318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:03.0340407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:03.0341024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:03.3959095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:03.4060275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:03.4061197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:03.4061709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:03.4062244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:03.4062863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:03.4069696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:03.4070693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:03.5073324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:03.8247792Z skip: Need at least 3 CUDA devices (2.078s) 2022-08-17T13:41:03.8248060Z 2022-08-17T13:41:03.8248376Z ---------------------------------------------------------------------- 2022-08-17T13:41:03.8248639Z Ran 1 test in 2.078s 2022-08-17T13:41:03.8248759Z 2022-08-17T13:41:03.8249058Z OK (skipped=1) 2022-08-17T13:41:03.8249176Z 2022-08-17T13:41:03.8249254Z Generating XML reports... 2022-08-17T13:41:03.8283820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134101.xml 2022-08-17T13:41:05.1495798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:05.1496207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:05.1498488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:05.1498878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:05.4950112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:05.4959503Z 2022-08-17T13:41:05.4959636Z Running tests... 2022-08-17T13:41:05.4960295Z ---------------------------------------------------------------------- 2022-08-17T13:41:05.7634757Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23842 2022-08-17T13:41:05.7663526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23843 2022-08-17T13:41:05.7692034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23844 2022-08-17T13:41:06.7849568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:06.7850235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:06.7851782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:06.7852462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:06.7874228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:06.7874839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:06.7875686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:06.7876312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:06.7878910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:06.7879529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:06.7880294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:06.7881186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:07.1692540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:07.1793157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:07.1793823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:07.1794831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:07.1795612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:07.1796481Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:07.1900565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:07.2806791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:07.2807532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:07.5740435Z skip: Need at least 3 CUDA devices (2.078s) 2022-08-17T13:41:07.5740778Z 2022-08-17T13:41:07.5741276Z ---------------------------------------------------------------------- 2022-08-17T13:41:07.5741579Z Ran 1 test in 2.078s 2022-08-17T13:41:07.5741707Z 2022-08-17T13:41:07.5741771Z OK (skipped=1) 2022-08-17T13:41:07.5741899Z 2022-08-17T13:41:07.5742045Z Generating XML reports... 2022-08-17T13:41:07.5778651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134105.xml 2022-08-17T13:41:08.8803735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:08.8804157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:08.8806750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:08.8807125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:09.2236687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:09.2246649Z 2022-08-17T13:41:09.2247014Z Running tests... 2022-08-17T13:41:09.2247440Z ---------------------------------------------------------------------- 2022-08-17T13:41:09.4894233Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23895 2022-08-17T13:41:09.4921734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23896 2022-08-17T13:41:09.4949409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23897 2022-08-17T13:41:10.4810675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:10.4811313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:10.4814340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:10.4815018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:10.4886743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:10.4887387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:10.4890305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:10.4890952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:10.5042539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:10.5043196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:10.5045549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:10.5046385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:10.8640913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:10.8741707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:10.8742333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:10.8743203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:10.8743948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:10.8744836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:10.8849212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:10.9756445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:10.9756998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:11.2997118Z skip: Need at least 3 CUDA devices (2.075s) 2022-08-17T13:41:11.2997354Z 2022-08-17T13:41:11.2997745Z ---------------------------------------------------------------------- 2022-08-17T13:41:11.2998066Z Ran 1 test in 2.075s 2022-08-17T13:41:11.2998185Z 2022-08-17T13:41:11.2998252Z OK (skipped=1) 2022-08-17T13:41:11.2998383Z 2022-08-17T13:41:11.2998518Z Generating XML reports... 2022-08-17T13:41:11.3034295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134109.xml 2022-08-17T13:41:12.5907179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:12.5907560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:12.5910023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:12.5910398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:12.9331361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:12.9341546Z 2022-08-17T13:41:12.9341635Z Running tests... 2022-08-17T13:41:12.9342696Z ---------------------------------------------------------------------- 2022-08-17T13:41:13.1996426Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23948 2022-08-17T13:41:13.2025085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23949 2022-08-17T13:41:13.2053467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23950 2022-08-17T13:41:14.1881374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:14.1881983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:14.1884401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:14.1885006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:14.1898140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:14.1899054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:14.1903571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:14.1904159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:14.2034761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:14.2035148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:14.2039203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:14.2039733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:14.5671980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:14.5772939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:14.5773439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:14.5774107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:14.5775088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:14.5776012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:14.5780835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:14.5783292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:14.5783886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:14.9100115Z skip: Need at least 3 CUDA devices (1.976s) 2022-08-17T13:41:14.9100423Z 2022-08-17T13:41:14.9100797Z ---------------------------------------------------------------------- 2022-08-17T13:41:14.9101138Z Ran 1 test in 1.976s 2022-08-17T13:41:14.9101265Z 2022-08-17T13:41:14.9101344Z OK (skipped=1) 2022-08-17T13:41:14.9101458Z 2022-08-17T13:41:14.9101561Z Generating XML reports... 2022-08-17T13:41:14.9136614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134112.xml 2022-08-17T13:41:16.1763750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:16.1764145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:16.1765749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:16.1766289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:16.5260419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:16.5271249Z 2022-08-17T13:41:16.5271560Z Running tests... 2022-08-17T13:41:16.5272192Z ---------------------------------------------------------------------- 2022-08-17T13:41:16.7952025Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24001 2022-08-17T13:41:16.7981055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24002 2022-08-17T13:41:16.8010348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24003 2022-08-17T13:41:17.7668036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:17.7668707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:17.7670418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:17.7671347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:17.7708421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:17.7709087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:17.7712977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:17.7713634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:17.8205123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:17.8205769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:17.8207623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:17.8208226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:18.1729033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:18.1830005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:18.1830504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:18.1831244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:18.1831792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:18.1832347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:18.1936396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:18.2844162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:18.2844753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:18.6061252Z skip: Need at least 3 CUDA devices (2.079s) 2022-08-17T13:41:18.6061615Z 2022-08-17T13:41:18.6062162Z ---------------------------------------------------------------------- 2022-08-17T13:41:18.6062469Z Ran 1 test in 2.079s 2022-08-17T13:41:18.6062660Z 2022-08-17T13:41:18.6062738Z OK (skipped=1) 2022-08-17T13:41:18.6062839Z 2022-08-17T13:41:18.6062929Z Generating XML reports... 2022-08-17T13:41:18.6097351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134116.xml 2022-08-17T13:41:19.8923024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:19.8923442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:19.8925730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:19.8926317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:20.2296486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:20.2307227Z 2022-08-17T13:41:20.2307502Z Running tests... 2022-08-17T13:41:20.2307969Z ---------------------------------------------------------------------- 2022-08-17T13:41:20.4917800Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24054 2022-08-17T13:41:20.4946168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24055 2022-08-17T13:41:20.4976065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24056 2022-08-17T13:41:21.4680394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:21.4681044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:21.4682439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:21.4683032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:21.4833732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:21.4834414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:21.4837533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:21.4837934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:21.5148937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:21.5149622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:21.5153667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:21.5154336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:21.8809203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:21.8810092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:21.8811081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:21.8811698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:21.8812236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:21.8812785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:21.8914391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:21.9822043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:21.9822557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:22.3025780Z skip: Need at least 2 CUDA devices (2.071s) 2022-08-17T13:41:22.3026066Z 2022-08-17T13:41:22.3026579Z ---------------------------------------------------------------------- 2022-08-17T13:41:22.3027043Z Ran 1 test in 2.072s 2022-08-17T13:41:22.3027237Z 2022-08-17T13:41:22.3027374Z OK (skipped=1) 2022-08-17T13:41:22.3027591Z 2022-08-17T13:41:22.3027753Z Generating XML reports... 2022-08-17T13:41:22.3065381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134120.xml 2022-08-17T13:41:23.6021439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:23.6022117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:23.6024541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:23.6025066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:23.9404777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:23.9414929Z 2022-08-17T13:41:23.9415058Z Running tests... 2022-08-17T13:41:23.9416047Z ---------------------------------------------------------------------- 2022-08-17T13:41:24.2047853Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24107 2022-08-17T13:41:24.2075993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24108 2022-08-17T13:41:24.2103890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24109 2022-08-17T13:41:25.2406736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:25.2407126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:25.2409945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:25.2410621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:25.2411474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:25.2411976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:25.2415483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:25.2416194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:25.2638638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:25.2639080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:25.2642370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:25.2642877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:25.6239760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:25.6340411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:25.6340879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:25.6341557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:25.6342120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:25.6342749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:25.6447935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:25.7354396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:25.7355071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:26.0154254Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T13:41:26.0154586Z 2022-08-17T13:41:26.0155121Z ---------------------------------------------------------------------- 2022-08-17T13:41:26.0155388Z Ran 1 test in 2.074s 2022-08-17T13:41:26.0155511Z 2022-08-17T13:41:26.0155588Z OK (skipped=1) 2022-08-17T13:41:26.0155717Z 2022-08-17T13:41:26.0155807Z Generating XML reports... 2022-08-17T13:41:26.0191273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134123.xml 2022-08-17T13:41:27.3172556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:27.3172942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:27.3175593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:27.3176110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:27.6613704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:27.6623350Z 2022-08-17T13:41:27.6623621Z Running tests... 2022-08-17T13:41:27.6624239Z ---------------------------------------------------------------------- 2022-08-17T13:41:27.9289881Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24160 2022-08-17T13:41:27.9319074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24161 2022-08-17T13:41:27.9347855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24162 2022-08-17T13:41:28.9324304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:28.9324953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:28.9327760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:28.9328417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:28.9350122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:28.9350767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:28.9354007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:28.9354622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:28.9419253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:28.9419889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:28.9423596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:28.9435136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:29.3036649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:29.3137250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:29.3137696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:29.3138360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:29.3138907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:29.3139461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:29.3246287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:29.4151357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:29.4152458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:29.7399853Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T13:41:29.7400067Z 2022-08-17T13:41:29.7400413Z ---------------------------------------------------------------------- 2022-08-17T13:41:29.7400774Z Ran 1 test in 2.078s 2022-08-17T13:41:29.7400896Z 2022-08-17T13:41:29.7400975Z OK (skipped=1) 2022-08-17T13:41:29.7401089Z 2022-08-17T13:41:29.7401166Z Generating XML reports... 2022-08-17T13:41:29.7436624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134127.xml 2022-08-17T13:41:31.0387401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:31.0388394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:31.0390685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:31.0391319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:31.3817787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:31.3828552Z 2022-08-17T13:41:31.3828978Z Running tests... 2022-08-17T13:41:31.3829411Z ---------------------------------------------------------------------- 2022-08-17T13:41:31.6494291Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24213 2022-08-17T13:41:31.6521937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24214 2022-08-17T13:41:31.6550572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24215 2022-08-17T13:41:32.6352044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:32.6352946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:32.6354431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:32.6355075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:32.6382701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:32.6383402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:32.6386684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:32.6387329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:32.6626750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:32.6627163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:32.6630323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:32.6630695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:33.0248553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:33.0349746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:33.0350335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:33.0351006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:33.0351566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:33.0352126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:33.0456746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:33.1362611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:33.1363223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:33.4599970Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T13:41:33.4600295Z 2022-08-17T13:41:33.4600737Z ---------------------------------------------------------------------- 2022-08-17T13:41:33.4601226Z Ran 1 test in 2.077s 2022-08-17T13:41:33.4601347Z 2022-08-17T13:41:33.4601425Z OK (skipped=1) 2022-08-17T13:41:33.4601542Z 2022-08-17T13:41:33.4601635Z Generating XML reports... 2022-08-17T13:41:33.4636674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134131.xml 2022-08-17T13:41:34.7411305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:34.7411702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:34.7414276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:34.7414663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:35.0863682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:35.0872647Z 2022-08-17T13:41:35.0872729Z Running tests... 2022-08-17T13:41:35.0873978Z ---------------------------------------------------------------------- 2022-08-17T13:41:35.3497167Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24266 2022-08-17T13:41:35.3525264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24267 2022-08-17T13:41:35.3553768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24268 2022-08-17T13:41:36.3358696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:36.3359378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:36.3363251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:36.3363916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:36.3453305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:36.3453927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:36.3456991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:36.3457657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:36.3608693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:36.3609338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:36.3612923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:36.3613334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:36.7209940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:36.7210667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:36.7211091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:36.7211742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:36.7212289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:36.7212840Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:36.7316755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:36.8222923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:36.8223816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:37.1601072Z skip: Need at least 2 CUDA devices (2.072s) 2022-08-17T13:41:37.1601352Z 2022-08-17T13:41:37.1601848Z ---------------------------------------------------------------------- 2022-08-17T13:41:37.1602251Z Ran 1 test in 2.073s 2022-08-17T13:41:37.1602450Z 2022-08-17T13:41:37.1602578Z OK (skipped=1) 2022-08-17T13:41:37.1602763Z 2022-08-17T13:41:37.1602888Z Generating XML reports... 2022-08-17T13:41:37.1638954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134135.xml 2022-08-17T13:41:38.4793037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:38.4793436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:38.4795655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:38.4796191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:38.8197509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:38.8208190Z 2022-08-17T13:41:38.8208371Z Running tests... 2022-08-17T13:41:38.8208997Z ---------------------------------------------------------------------- 2022-08-17T13:41:39.0869099Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24319 2022-08-17T13:41:39.0896176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24320 2022-08-17T13:41:39.0926529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24321 2022-08-17T13:41:40.0604021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:40.0604723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:40.0607887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:40.0608518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:40.0926469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:40.0927132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:40.0930454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:40.0931162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:40.0974905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:40.0975543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:40.0979479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:40.0980163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:40.4546972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:40.4648794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:40.4649562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:40.4650261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:40.4650809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:40.4651553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:40.4656789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:40.4658627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:40.4659496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:40.7977357Z skip: Need at least 2 CUDA devices (1.977s) 2022-08-17T13:41:40.7977658Z 2022-08-17T13:41:40.7978132Z ---------------------------------------------------------------------- 2022-08-17T13:41:40.7978555Z Ran 1 test in 1.977s 2022-08-17T13:41:40.7978746Z 2022-08-17T13:41:40.7978846Z OK (skipped=1) 2022-08-17T13:41:40.7979021Z 2022-08-17T13:41:40.7980068Z Generating XML reports... 2022-08-17T13:41:40.8014760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134138.xml 2022-08-17T13:41:42.0657562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:42.0658520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:42.0660619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:42.0661240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:42.4138960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:42.4148892Z 2022-08-17T13:41:42.4149171Z Running tests... 2022-08-17T13:41:42.4149803Z ---------------------------------------------------------------------- 2022-08-17T13:41:42.6825554Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24372 2022-08-17T13:41:42.6854798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24373 2022-08-17T13:41:42.6883197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24374 2022-08-17T13:41:43.6668018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:43.6668647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:43.6671204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:43.6703747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:43.6704327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:43.6704687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:43.6708196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:43.6708580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:43.6970215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:43.6970678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:43.6974386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:43.6974830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:44.0648039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:44.0748529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:44.0749530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:44.0750619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:44.0751283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:44.0751831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:44.0757251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:44.0758248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:44.0758669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:44.3933238Z skip: Need at least 2 CUDA devices (1.978s) 2022-08-17T13:41:44.3933469Z 2022-08-17T13:41:44.3933923Z ---------------------------------------------------------------------- 2022-08-17T13:41:44.3934192Z Ran 1 test in 1.978s 2022-08-17T13:41:44.3934315Z 2022-08-17T13:41:44.3934378Z OK (skipped=1) 2022-08-17T13:41:44.3934492Z 2022-08-17T13:41:44.3934763Z Generating XML reports... 2022-08-17T13:41:44.3969770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134142.xml 2022-08-17T13:41:45.6755602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:45.6756004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:45.6758405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:45.6758932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:46.0173126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:46.0182845Z 2022-08-17T13:41:46.0183005Z Running tests... 2022-08-17T13:41:46.0183633Z ---------------------------------------------------------------------- 2022-08-17T13:41:46.2859626Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24425 2022-08-17T13:41:46.2893251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24426 2022-08-17T13:41:46.2924949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24427 2022-08-17T13:41:47.2747531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:47.2748222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:47.2750952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:47.2751571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:47.2877079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:47.2877701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:47.2880763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:47.2881176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:47.3358364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:47.3358786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:47.3361866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:47.3362616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:47.6952162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:47.7031680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:47.7032427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:47.7033102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:47.7033665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:47.7053883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:47.7139679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:47.7140412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:47.8067506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:48.0974835Z skip: Need at least 2 CUDA devices (2.079s) 2022-08-17T13:41:48.0975196Z 2022-08-17T13:41:48.0975631Z ---------------------------------------------------------------------- 2022-08-17T13:41:48.0975903Z Ran 1 test in 2.079s 2022-08-17T13:41:48.0976026Z 2022-08-17T13:41:48.0976103Z OK (skipped=1) 2022-08-17T13:41:48.0976217Z 2022-08-17T13:41:48.0976307Z Generating XML reports... 2022-08-17T13:41:48.1013512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134146.xml 2022-08-17T13:41:49.4012769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:49.4013183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:49.4015486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:49.4015877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:49.7458806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:49.7468809Z 2022-08-17T13:41:49.7468966Z Running tests... 2022-08-17T13:41:49.7469391Z ---------------------------------------------------------------------- 2022-08-17T13:41:50.0123552Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24478 2022-08-17T13:41:50.0153720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24479 2022-08-17T13:41:50.0184904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24480 2022-08-17T13:41:51.0298117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:51.0298526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:51.0301265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:51.0301712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:51.0701849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:51.0702537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:51.0706042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:51.0706962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:51.0785566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:51.0786156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:51.0790712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:51.0791371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:51.4469317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:51.4499045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:51.4499808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:51.4500543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:51.4501127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:51.4571625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:51.4607985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:51.4608547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:51.5584146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:51.8232443Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:41:51.8232713Z 2022-08-17T13:41:51.8233119Z ---------------------------------------------------------------------- 2022-08-17T13:41:51.8233444Z Ran 1 test in 2.076s 2022-08-17T13:41:51.8233583Z 2022-08-17T13:41:51.8233662Z OK (skipped=1) 2022-08-17T13:41:51.8233761Z 2022-08-17T13:41:51.8233855Z Generating XML reports... 2022-08-17T13:41:51.8268616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134149.xml 2022-08-17T13:41:53.1359922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:53.1360321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:53.1362764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:53.1363143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:53.4743018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:53.4753518Z 2022-08-17T13:41:53.4753835Z Running tests... 2022-08-17T13:41:53.4754501Z ---------------------------------------------------------------------- 2022-08-17T13:41:53.7432518Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24531 2022-08-17T13:41:53.7460540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24532 2022-08-17T13:41:53.7489634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24533 2022-08-17T13:41:54.7169144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:54.7169772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:54.7172067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:54.7172700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:54.7585840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:54.7586523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:54.7589557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:54.7590176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:54.7622159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:54.7622901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:54.7626932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:54.7627583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:55.1353485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:55.1455228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:55.1456136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:55.1456910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:55.1457782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:55.1458512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:55.1464087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:55.1465114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:55.1466764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:55.4535907Z skip: Need at least 2 CUDA devices (1.978s) 2022-08-17T13:41:55.4536170Z 2022-08-17T13:41:55.4536745Z ---------------------------------------------------------------------- 2022-08-17T13:41:55.4537054Z Ran 1 test in 1.978s 2022-08-17T13:41:55.4537175Z 2022-08-17T13:41:55.4537252Z OK (skipped=1) 2022-08-17T13:41:55.4537366Z 2022-08-17T13:41:55.4537443Z Generating XML reports... 2022-08-17T13:41:55.4572025Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134153.xml 2022-08-17T13:41:56.7292875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:56.7293328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:56.7295624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:56.7296003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:57.0714835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:41:57.0724572Z 2022-08-17T13:41:57.0724747Z Running tests... 2022-08-17T13:41:57.0725113Z ---------------------------------------------------------------------- 2022-08-17T13:41:57.3366585Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24584 2022-08-17T13:41:57.3395719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24585 2022-08-17T13:41:57.3423683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24586 2022-08-17T13:41:58.3018743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:58.3019482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:58.3022146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:58.3022531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:58.3070742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:58.3071381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:58.3074528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:58.3075187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:58.3493820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:41:58.3497139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:41:58.3497878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:41:58.3498410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:41:58.7225985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:41:58.7328205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:41:58.7328833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:41:58.7329756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:58.7330327Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:58.7429406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:41:58.7438028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:41:58.7461732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:41:58.8442439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:41:59.1470613Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T13:41:59.1470917Z 2022-08-17T13:41:59.1471453Z ---------------------------------------------------------------------- 2022-08-17T13:41:59.1471805Z Ran 1 test in 2.075s 2022-08-17T13:41:59.1471925Z 2022-08-17T13:41:59.1472001Z OK (skipped=1) 2022-08-17T13:41:59.1472117Z 2022-08-17T13:41:59.1472207Z Generating XML reports... 2022-08-17T13:41:59.1507327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134157.xml 2022-08-17T13:42:00.4443918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:00.4444323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:00.4446426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:00.4446849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:00.7952605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:00.7966768Z 2022-08-17T13:42:00.7966918Z Running tests... 2022-08-17T13:42:00.7967515Z ---------------------------------------------------------------------- 2022-08-17T13:42:01.0687178Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24637 2022-08-17T13:42:01.0715693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24638 2022-08-17T13:42:01.0743533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24639 2022-08-17T13:42:02.0796952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:02.0797360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:02.0800055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:02.0800438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:02.1014291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:02.1014654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:02.1018250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:02.1018647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:02.1153400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:02.1154015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:02.1157956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:02.1158322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:02.4769696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:02.4862669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:02.4863483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:02.4864470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:02.4865032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:02.4871219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:02.4970735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:02.4971449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:02.5883607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:02.8790503Z skip: Need at least 2 CUDA devices (2.082s) 2022-08-17T13:42:02.8790807Z 2022-08-17T13:42:02.8791295Z ---------------------------------------------------------------------- 2022-08-17T13:42:02.8791568Z Ran 1 test in 2.082s 2022-08-17T13:42:02.8791690Z 2022-08-17T13:42:02.8791766Z OK (skipped=1) 2022-08-17T13:42:02.8791879Z 2022-08-17T13:42:02.8791956Z Generating XML reports... 2022-08-17T13:42:02.8826121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134200.xml 2022-08-17T13:42:04.1774648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:04.1775134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:04.1776375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:04.1776745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:04.5164061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:04.5173463Z 2022-08-17T13:42:04.5173857Z Running tests... 2022-08-17T13:42:04.5174515Z ---------------------------------------------------------------------- 2022-08-17T13:42:04.5192121Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:42:04.5192629Z 2022-08-17T13:42:04.5193046Z ---------------------------------------------------------------------- 2022-08-17T13:42:04.5193312Z Ran 1 test in 0.002s 2022-08-17T13:42:04.5193436Z 2022-08-17T13:42:04.5193514Z OK (skipped=1) 2022-08-17T13:42:04.5193629Z 2022-08-17T13:42:04.5193706Z Generating XML reports... 2022-08-17T13:42:04.5217843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134204.xml 2022-08-17T13:42:05.6782159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:05.6782550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:05.6786016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:05.6786416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:06.0211824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:06.0220935Z 2022-08-17T13:42:06.0221042Z Running tests... 2022-08-17T13:42:06.0221469Z ---------------------------------------------------------------------- 2022-08-17T13:42:06.2860764Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24700 2022-08-17T13:42:06.2889578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24701 2022-08-17T13:42:06.2918012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24702 2022-08-17T13:42:07.2804270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:07.2804957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:07.2806796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:07.2807479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:07.2885084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:07.2889035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:07.2889850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:07.2890501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:07.3087048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:07.3087758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:07.3090677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:07.3091307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:07.6892343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:07.6892887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:07.6893276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:07.6893916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:07.6894473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:07.6895188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:07.6900483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:07.6901976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:07.6904154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:07.9965177Z skip: Need at least 2 CUDA devices (1.974s) 2022-08-17T13:42:07.9965543Z 2022-08-17T13:42:07.9966142Z ---------------------------------------------------------------------- 2022-08-17T13:42:07.9966408Z Ran 1 test in 1.974s 2022-08-17T13:42:07.9966531Z 2022-08-17T13:42:07.9966607Z OK (skipped=1) 2022-08-17T13:42:07.9966707Z 2022-08-17T13:42:07.9966797Z Generating XML reports... 2022-08-17T13:42:08.0001396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134206.xml 2022-08-17T13:42:09.2976137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:09.2976528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:09.2979368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:09.2979754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:09.6392085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:09.6401691Z 2022-08-17T13:42:09.6401842Z Running tests... 2022-08-17T13:42:09.6402304Z ---------------------------------------------------------------------- 2022-08-17T13:42:09.9059890Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24753 2022-08-17T13:42:09.9088860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24754 2022-08-17T13:42:09.9116699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24755 2022-08-17T13:42:10.9424739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:10.9425445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:10.9426172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:10.9426774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:10.9427997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:10.9428604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:10.9429362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:10.9430008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:10.9678276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:10.9678959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:10.9682324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:10.9682747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:11.3358424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:11.3459648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:11.3460298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:11.3461420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:11.3461994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:11.3463015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:11.3468630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:11.3469268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:11.3469840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:11.7166638Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:42:11.7166983Z 2022-08-17T13:42:11.7167468Z ---------------------------------------------------------------------- 2022-08-17T13:42:11.7167744Z Ran 1 test in 2.076s 2022-08-17T13:42:11.7167850Z 2022-08-17T13:42:11.7167927Z OK (skipped=1) 2022-08-17T13:42:11.7168038Z 2022-08-17T13:42:11.7168127Z Generating XML reports... 2022-08-17T13:42:11.7205697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134209.xml 2022-08-17T13:42:12.9987319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:12.9987712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:12.9990273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:12.9990751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:13.3475904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:13.3485456Z 2022-08-17T13:42:13.3485597Z Running tests... 2022-08-17T13:42:13.3486234Z ---------------------------------------------------------------------- 2022-08-17T13:42:13.6190339Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24806 2022-08-17T13:42:13.6219349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24807 2022-08-17T13:42:13.6247727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24808 2022-08-17T13:42:14.6597662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:14.6598324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:14.6599762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:14.6600376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:14.6680159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:14.6680844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:14.6684041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:14.6684708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:14.6688649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:14.6689256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:14.6693103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:14.6693776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:15.0314112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:15.0360407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:15.0360857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:15.0361523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:15.0362089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:15.0415015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:15.0468410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:15.0469039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:15.0539119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinya0u_2 2022-08-17T13:42:15.0539830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uw6k1fq 2022-08-17T13:42:15.0541114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinya0u_2/_remote_module_non_scriptable.py 2022-08-17T13:42:15.0541792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uw6k1fq/_remote_module_non_scriptable.py 2022-08-17T13:42:15.1424960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:15.1495260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao6n_n9k 2022-08-17T13:42:15.1497215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao6n_n9k/_remote_module_non_scriptable.py 2022-08-17T13:42:15.1639780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:42:15.1640456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:42:15.1640839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-08-17T13:42:15.6300098Z ok (2.281s) 2022-08-17T13:42:15.6300294Z 2022-08-17T13:42:15.6300665Z ---------------------------------------------------------------------- 2022-08-17T13:42:15.6300908Z Ran 1 test in 2.281s 2022-08-17T13:42:15.6301037Z 2022-08-17T13:42:15.6301101Z OK 2022-08-17T13:42:15.6301195Z 2022-08-17T13:42:15.6301289Z Generating XML reports... 2022-08-17T13:42:15.6336034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134213.xml 2022-08-17T13:42:16.9238967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:16.9239457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:16.9241814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:16.9242192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:17.2682532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:17.2692397Z 2022-08-17T13:42:17.2692503Z Running tests... 2022-08-17T13:42:17.2693401Z ---------------------------------------------------------------------- 2022-08-17T13:42:17.5378516Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24868 2022-08-17T13:42:17.5406142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24869 2022-08-17T13:42:17.5434533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24870 2022-08-17T13:42:18.5071204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:18.5072135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:18.5112724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:18.5113434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:18.5186944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:18.5187620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:18.5191093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:18.5191735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:18.5192515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:18.5193200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:18.5196755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:18.5197347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:18.8828353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:18.8929232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:18.8929910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:18.8930569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:18.8931132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:18.8931684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:18.8937291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:18.8966659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:18.8967049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:19.2484015Z skip: CUDA is not available. (1.979s) 2022-08-17T13:42:19.2484338Z 2022-08-17T13:42:19.2484866Z ---------------------------------------------------------------------- 2022-08-17T13:42:19.2485188Z Ran 1 test in 1.979s 2022-08-17T13:42:19.2485308Z 2022-08-17T13:42:19.2485371Z OK (skipped=1) 2022-08-17T13:42:19.2485484Z 2022-08-17T13:42:19.2485574Z Generating XML reports... 2022-08-17T13:42:19.2520848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134217.xml 2022-08-17T13:42:20.5249869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:20.5250332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:20.5251984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:20.5252406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:20.8660678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:20.8670795Z 2022-08-17T13:42:20.8671103Z Running tests... 2022-08-17T13:42:20.8671780Z ---------------------------------------------------------------------- 2022-08-17T13:42:20.8693395Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:42:20.8694187Z 2022-08-17T13:42:20.8694463Z ---------------------------------------------------------------------- 2022-08-17T13:42:20.8694724Z Ran 1 test in 0.002s 2022-08-17T13:42:20.8694846Z 2022-08-17T13:42:20.8694923Z OK (skipped=1) 2022-08-17T13:42:20.8695022Z 2022-08-17T13:42:20.8695114Z Generating XML reports... 2022-08-17T13:42:20.8719047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134220.xml 2022-08-17T13:42:22.0095889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:22.0096284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:22.0099124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:22.0099531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:22.3448925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:22.3458671Z 2022-08-17T13:42:22.3458833Z Running tests... 2022-08-17T13:42:22.3459224Z ---------------------------------------------------------------------- 2022-08-17T13:42:22.3481401Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:42:22.3481704Z 2022-08-17T13:42:22.3481925Z ---------------------------------------------------------------------- 2022-08-17T13:42:22.3482246Z Ran 1 test in 0.002s 2022-08-17T13:42:22.3482375Z 2022-08-17T13:42:22.3482441Z OK (skipped=1) 2022-08-17T13:42:22.3482555Z 2022-08-17T13:42:22.3482643Z Generating XML reports... 2022-08-17T13:42:22.3507923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134222.xml 2022-08-17T13:42:23.5014120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:23.5014617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:23.5016643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:23.5017027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:23.8458414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:23.8468662Z 2022-08-17T13:42:23.8469320Z Running tests... 2022-08-17T13:42:23.8469762Z ---------------------------------------------------------------------- 2022-08-17T13:42:23.8489334Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:42:23.8489789Z 2022-08-17T13:42:23.8490010Z ---------------------------------------------------------------------- 2022-08-17T13:42:23.8490287Z Ran 1 test in 0.002s 2022-08-17T13:42:23.8490395Z 2022-08-17T13:42:23.8490470Z OK (skipped=1) 2022-08-17T13:42:23.8490582Z 2022-08-17T13:42:23.8490671Z Generating XML reports... 2022-08-17T13:42:23.8514582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134223.xml 2022-08-17T13:42:24.9966654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:24.9967289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:24.9969530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:24.9970114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:25.3379315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:25.3389615Z 2022-08-17T13:42:25.3390040Z Running tests... 2022-08-17T13:42:25.3390724Z ---------------------------------------------------------------------- 2022-08-17T13:42:25.3407840Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:42:25.3408355Z 2022-08-17T13:42:25.3408761Z ---------------------------------------------------------------------- 2022-08-17T13:42:25.3409027Z Ran 1 test in 0.002s 2022-08-17T13:42:25.3409148Z 2022-08-17T13:42:25.3409227Z OK (skipped=1) 2022-08-17T13:42:25.3409327Z 2022-08-17T13:42:25.3409418Z Generating XML reports... 2022-08-17T13:42:25.3433868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134225.xml 2022-08-17T13:42:26.4992915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:26.4993310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:26.4995860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:26.4996342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:26.8417590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:26.8426544Z 2022-08-17T13:42:26.8426652Z Running tests... 2022-08-17T13:42:26.8427266Z ---------------------------------------------------------------------- 2022-08-17T13:42:26.8453430Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:42:26.8453935Z 2022-08-17T13:42:26.8454227Z ---------------------------------------------------------------------- 2022-08-17T13:42:26.8454710Z Ran 1 test in 0.003s 2022-08-17T13:42:26.8454932Z 2022-08-17T13:42:26.8455073Z OK (skipped=1) 2022-08-17T13:42:26.8455273Z 2022-08-17T13:42:26.8455385Z Generating XML reports... 2022-08-17T13:42:26.8479189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134226.xml 2022-08-17T13:42:27.9991494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:27.9991893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:27.9994563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:27.9995146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:28.3517066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:28.3527010Z 2022-08-17T13:42:28.3527124Z Running tests... 2022-08-17T13:42:28.3527735Z ---------------------------------------------------------------------- 2022-08-17T13:42:28.6244064Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24971 2022-08-17T13:42:28.6275694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24972 2022-08-17T13:42:28.6305556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24973 2022-08-17T13:42:29.6244049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:29.6244469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:29.6247285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:29.6247766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:29.6383125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:29.6383669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:29.6387199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:29.6387575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:29.6442189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:29.6442553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:29.6446892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:29.6447271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:30.0214371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:30.0314643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:30.0315372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:30.0316175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:30.0317294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:30.0317868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:30.0424283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:30.0424787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:30.1329911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:30.4351819Z skip: Need at least 2 CUDA devices (2.082s) 2022-08-17T13:42:30.4352096Z 2022-08-17T13:42:30.4352585Z ---------------------------------------------------------------------- 2022-08-17T13:42:30.4352848Z Ran 1 test in 2.082s 2022-08-17T13:42:30.4352959Z 2022-08-17T13:42:30.4353038Z OK (skipped=1) 2022-08-17T13:42:30.4353238Z 2022-08-17T13:42:30.4353331Z Generating XML reports... 2022-08-17T13:42:30.4388691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134228.xml 2022-08-17T13:42:31.7599194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:31.7599593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:31.7602002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:31.7602366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:32.1071015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:32.1080878Z 2022-08-17T13:42:32.1081026Z Running tests... 2022-08-17T13:42:32.1081432Z ---------------------------------------------------------------------- 2022-08-17T13:42:32.3707361Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25024 2022-08-17T13:42:32.3734248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25025 2022-08-17T13:42:32.3762786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25026 2022-08-17T13:42:33.3603618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:33.3603998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:33.3606896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:33.3607602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:33.3815106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:33.3815759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:33.3818531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:33.3819158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:33.3975493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:33.3976179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:33.3979952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:33.3980581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:33.7550921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:33.7651543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:33.7652152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:33.7652792Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:33.7653354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:33.7653909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:33.7758418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:33.8665034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:34.1810602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:34.1811280Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:42:34.1811527Z 2022-08-17T13:42:34.1811991Z ---------------------------------------------------------------------- 2022-08-17T13:42:34.1812415Z Ran 1 test in 2.073s 2022-08-17T13:42:34.1812601Z 2022-08-17T13:42:34.1812724Z OK (skipped=1) 2022-08-17T13:42:34.1812920Z 2022-08-17T13:42:34.1813065Z Generating XML reports... 2022-08-17T13:42:34.1847917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134232.xml 2022-08-17T13:42:35.4673511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:35.4673918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:35.4675679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:35.4676041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:35.8139133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:35.8149210Z 2022-08-17T13:42:35.8149324Z Running tests... 2022-08-17T13:42:35.8149937Z ---------------------------------------------------------------------- 2022-08-17T13:42:35.8168808Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:42:35.8169313Z 2022-08-17T13:42:35.8169698Z ---------------------------------------------------------------------- 2022-08-17T13:42:35.8170157Z Ran 1 test in 0.002s 2022-08-17T13:42:35.8170372Z 2022-08-17T13:42:35.8170501Z OK (skipped=1) 2022-08-17T13:42:35.8170994Z 2022-08-17T13:42:35.8171136Z Generating XML reports... 2022-08-17T13:42:35.8195650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134235.xml 2022-08-17T13:42:36.9734640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:36.9735020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:36.9737718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:36.9738103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:37.3124714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:37.3135352Z 2022-08-17T13:42:37.3135704Z Running tests... 2022-08-17T13:42:37.3136379Z ---------------------------------------------------------------------- 2022-08-17T13:42:37.3156237Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:42:37.3156708Z 2022-08-17T13:42:37.3157081Z ---------------------------------------------------------------------- 2022-08-17T13:42:37.3157476Z Ran 1 test in 0.002s 2022-08-17T13:42:37.3157940Z 2022-08-17T13:42:37.3158076Z OK (skipped=1) 2022-08-17T13:42:37.3158280Z 2022-08-17T13:42:37.3158439Z Generating XML reports... 2022-08-17T13:42:37.3183156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134237.xml 2022-08-17T13:42:38.4597592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:38.4598214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:38.4600040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:38.4600683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:38.8031256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:38.8041549Z 2022-08-17T13:42:38.8042021Z Running tests... 2022-08-17T13:42:38.8042474Z ---------------------------------------------------------------------- 2022-08-17T13:42:39.0686220Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25097 2022-08-17T13:42:39.0716300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25098 2022-08-17T13:42:39.0747008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25099 2022-08-17T13:42:40.0419164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:40.0419861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:40.0420729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:40.0421358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:40.0422184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:40.0422844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:40.0425714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:40.0426214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:40.0426839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:40.0427460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:40.0428471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:40.0429064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:40.4115725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:40.4217028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:40.4217674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:40.4218530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:40.4219448Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:40.4220251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:40.4228116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:40.4228907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:40.4229719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:40.7795620Z skip: Need at least 2 CUDA devices (1.975s) 2022-08-17T13:42:40.7795945Z 2022-08-17T13:42:40.7796401Z ---------------------------------------------------------------------- 2022-08-17T13:42:40.7796799Z Ran 1 test in 1.975s 2022-08-17T13:42:40.7796985Z 2022-08-17T13:42:40.7797099Z OK (skipped=1) 2022-08-17T13:42:40.7797276Z 2022-08-17T13:42:40.7797404Z Generating XML reports... 2022-08-17T13:42:40.7835939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134238.xml 2022-08-17T13:42:42.0871092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:42.0871487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:42.0873982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:42.0874350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:42.4282198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:42.4292721Z 2022-08-17T13:42:42.4292819Z Running tests... 2022-08-17T13:42:42.4293971Z ---------------------------------------------------------------------- 2022-08-17T13:42:42.6921908Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25150 2022-08-17T13:42:42.6952362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25151 2022-08-17T13:42:42.6980751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25152 2022-08-17T13:42:43.6802552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:43.6802965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:43.6806061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:43.6806736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:43.6888746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:43.6889401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:43.6892561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:43.6893492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:43.6980683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:43.6981337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:43.6984657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:43.6985317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:44.0560490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:44.0660237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:44.0660998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:44.0661989Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:44.0662919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:44.0664003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:44.0761856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:44.1676417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:44.1676833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:44.5028415Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:42:44.5028602Z 2022-08-17T13:42:44.5028914Z ---------------------------------------------------------------------- 2022-08-17T13:42:44.5029190Z Ran 1 test in 2.073s 2022-08-17T13:42:44.5029327Z 2022-08-17T13:42:44.5029404Z OK (skipped=1) 2022-08-17T13:42:44.5029518Z 2022-08-17T13:42:44.5029608Z Generating XML reports... 2022-08-17T13:42:44.5068460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134242.xml 2022-08-17T13:42:45.8067000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:45.8067404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:45.8069992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:45.8070379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:46.1499308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:46.1509372Z 2022-08-17T13:42:46.1509527Z Running tests... 2022-08-17T13:42:46.1510686Z ---------------------------------------------------------------------- 2022-08-17T13:42:46.1536119Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:42:46.1536463Z 2022-08-17T13:42:46.1536802Z ---------------------------------------------------------------------- 2022-08-17T13:42:46.1537108Z Ran 1 test in 0.003s 2022-08-17T13:42:46.1537235Z 2022-08-17T13:42:46.1537311Z OK (skipped=1) 2022-08-17T13:42:46.1537424Z 2022-08-17T13:42:46.1537512Z Generating XML reports... 2022-08-17T13:42:46.1562346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134246.xml 2022-08-17T13:42:47.3221193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:47.3221615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:47.3224088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:47.3224724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:47.6617387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:47.6627692Z 2022-08-17T13:42:47.6628102Z Running tests... 2022-08-17T13:42:47.6628514Z ---------------------------------------------------------------------- 2022-08-17T13:42:47.9250074Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25213 2022-08-17T13:42:47.9278434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25214 2022-08-17T13:42:47.9306410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25215 2022-08-17T13:42:48.9256814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:48.9257510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:48.9260183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:48.9260894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:48.9277006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:48.9277660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:48.9282441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:48.9283101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:48.9326194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:48.9326833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:48.9330111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:48.9330787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:49.3099658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:49.3201280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:49.3201815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:49.3202479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:49.3203064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:49.3203749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:49.3210619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:49.3211335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:49.3212340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:49.6354877Z skip: Need at least 2 CUDA devices (1.972s) 2022-08-17T13:42:49.6355069Z 2022-08-17T13:42:49.6355386Z ---------------------------------------------------------------------- 2022-08-17T13:42:49.6355661Z Ran 1 test in 1.973s 2022-08-17T13:42:49.6355767Z 2022-08-17T13:42:49.6355845Z OK (skipped=1) 2022-08-17T13:42:49.6355957Z 2022-08-17T13:42:49.6356047Z Generating XML reports... 2022-08-17T13:42:49.6389644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134247.xml 2022-08-17T13:42:50.9269434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:50.9269863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:50.9272551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:50.9273071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:51.2720063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:51.2729381Z 2022-08-17T13:42:51.2729516Z Running tests... 2022-08-17T13:42:51.2730160Z ---------------------------------------------------------------------- 2022-08-17T13:42:51.5447169Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25266 2022-08-17T13:42:51.5475277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25267 2022-08-17T13:42:51.5504233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25268 2022-08-17T13:42:52.5641221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:52.5642213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:52.5644952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:52.5645532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:52.5871750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:52.5872377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:52.5874961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:52.5875604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:52.5892157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:52.5892867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:52.5896535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:52.5897207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:52.9636985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:52.9637658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:52.9638170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:52.9638854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:52.9639681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:52.9640659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:52.9645526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:52.9646359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:52.9647120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:53.2549648Z skip: Need at least 2 CUDA devices (1.982s) 2022-08-17T13:42:53.2549845Z 2022-08-17T13:42:53.2550165Z ---------------------------------------------------------------------- 2022-08-17T13:42:53.2550671Z Ran 1 test in 1.982s 2022-08-17T13:42:53.2550795Z 2022-08-17T13:42:53.2550872Z OK (skipped=1) 2022-08-17T13:42:53.2550975Z 2022-08-17T13:42:53.2551071Z Generating XML reports... 2022-08-17T13:42:53.2585578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134251.xml 2022-08-17T13:42:54.5255515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:54.5255894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:54.5258375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:54.5258761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:54.8677767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:54.8688139Z 2022-08-17T13:42:54.8688293Z Running tests... 2022-08-17T13:42:54.8688841Z ---------------------------------------------------------------------- 2022-08-17T13:42:55.1311200Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25319 2022-08-17T13:42:55.1338563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25320 2022-08-17T13:42:55.1365764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25321 2022-08-17T13:42:56.1374085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:56.1374477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:56.1377359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:56.1377753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:56.1385778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:56.1386280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:56.1390957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:56.1391676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:56.1396038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:56.1396654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:56.1401089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:56.1401762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:56.5094553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:42:56.5195832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:42:56.5196681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:56.5197239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:42:56.5198139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:56.5198861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:42:56.5203918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:42:56.5204640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:42:56.5205522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:42:56.8414129Z skip: Need at least 2 CUDA devices (1.972s) 2022-08-17T13:42:56.8414426Z 2022-08-17T13:42:56.8414899Z ---------------------------------------------------------------------- 2022-08-17T13:42:56.8415332Z Ran 1 test in 1.972s 2022-08-17T13:42:56.8415531Z 2022-08-17T13:42:56.8415646Z OK (skipped=1) 2022-08-17T13:42:56.8415806Z 2022-08-17T13:42:56.8415942Z Generating XML reports... 2022-08-17T13:42:56.8451423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134254.xml 2022-08-17T13:42:58.1120893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:58.1121292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:58.1123884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:58.1124279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:58.4519604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:42:58.4529602Z 2022-08-17T13:42:58.4529744Z Running tests... 2022-08-17T13:42:58.4530362Z ---------------------------------------------------------------------- 2022-08-17T13:42:58.7227984Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25372 2022-08-17T13:42:58.7256939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25373 2022-08-17T13:42:58.7285155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25374 2022-08-17T13:42:59.7081588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:59.7081991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:59.7085320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:59.7086179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:59.7147853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:59.7148514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:59.7151823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:59.7152328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:42:59.7638503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:42:59.7638943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:42:59.7667832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:42:59.7668494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:00.1250915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:00.1297925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:00.1298344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:00.1299005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:00.1299568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:00.1352098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:00.1406150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:00.1406741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:00.2364385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:00.5333476Z skip: Need at least 2 CUDA devices (2.080s) 2022-08-17T13:43:00.5333710Z 2022-08-17T13:43:00.5334059Z ---------------------------------------------------------------------- 2022-08-17T13:43:00.5334326Z Ran 1 test in 2.080s 2022-08-17T13:43:00.5334447Z 2022-08-17T13:43:00.5334525Z OK (skipped=1) 2022-08-17T13:43:00.5334642Z 2022-08-17T13:43:00.5334733Z Generating XML reports... 2022-08-17T13:43:00.5369829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134258.xml 2022-08-17T13:43:01.8213844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:01.8214535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:01.8216092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:01.8216750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:02.1692698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:02.1702628Z 2022-08-17T13:43:02.1702766Z Running tests... 2022-08-17T13:43:02.1703682Z ---------------------------------------------------------------------- 2022-08-17T13:43:02.4341620Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25425 2022-08-17T13:43:02.4369487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25426 2022-08-17T13:43:02.4397389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25427 2022-08-17T13:43:03.4621769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:03.4622542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:03.4623571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:03.4624216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:03.4633632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:03.4634288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:03.4638366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:03.4798297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:03.4799299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:03.4799888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:03.4801157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:03.4801793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:03.8499175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:03.8600034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:03.8600687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:03.8601675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:03.8602458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:03.8603309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:03.8609139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:03.8609682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:03.8611109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:04.2444597Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T13:43:04.2444916Z 2022-08-17T13:43:04.2445428Z ---------------------------------------------------------------------- 2022-08-17T13:43:04.2445758Z Ran 1 test in 2.074s 2022-08-17T13:43:04.2446026Z 2022-08-17T13:43:04.2446104Z OK (skipped=1) 2022-08-17T13:43:04.2446225Z 2022-08-17T13:43:04.2446313Z Generating XML reports... 2022-08-17T13:43:04.2481751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134302.xml 2022-08-17T13:43:05.5433625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:05.5434035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:05.5436135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:05.5436693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:05.8860768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:05.8870074Z 2022-08-17T13:43:05.8870231Z Running tests... 2022-08-17T13:43:05.8870844Z ---------------------------------------------------------------------- 2022-08-17T13:43:06.1624649Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25478 2022-08-17T13:43:06.1653348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25479 2022-08-17T13:43:06.1681955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25480 2022-08-17T13:43:07.1451353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:07.1451755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:07.1454911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:07.1455293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:07.1469244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:07.1470058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:07.1474638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:07.1475324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:07.1850293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:07.1850936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:07.1853449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:07.5368511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:07.5368905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:07.5453276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:07.5454043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:07.5454810Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:07.5455360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:07.5470596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:07.5561179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:07.5561886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:07.6482647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:07.9733669Z skip: Need at least 2 CUDA devices (2.086s) 2022-08-17T13:43:07.9733987Z 2022-08-17T13:43:07.9734478Z ---------------------------------------------------------------------- 2022-08-17T13:43:07.9734753Z Ran 1 test in 2.086s 2022-08-17T13:43:07.9734873Z 2022-08-17T13:43:07.9735164Z OK (skipped=1) 2022-08-17T13:43:07.9735280Z 2022-08-17T13:43:07.9735370Z Generating XML reports... 2022-08-17T13:43:07.9769878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134305.xml 2022-08-17T13:43:09.2866568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:09.2866957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:09.2869617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:09.2870179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:09.6344206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:09.6355065Z 2022-08-17T13:43:09.6355883Z Running tests... 2022-08-17T13:43:09.6356306Z ---------------------------------------------------------------------- 2022-08-17T13:43:09.8965391Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s) 2022-08-17T13:43:09.8966191Z 2022-08-17T13:43:09.8966413Z ---------------------------------------------------------------------- 2022-08-17T13:43:09.8966688Z Ran 1 test in 0.261s 2022-08-17T13:43:09.8966796Z 2022-08-17T13:43:09.8966879Z OK (skipped=1) 2022-08-17T13:43:09.8966993Z 2022-08-17T13:43:09.8967081Z Generating XML reports... 2022-08-17T13:43:09.8990107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134309.xml 2022-08-17T13:43:11.1590257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:11.1590976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:11.1593806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:11.1594400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:11.5058046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:11.5068308Z 2022-08-17T13:43:11.5068802Z Running tests... 2022-08-17T13:43:11.5069224Z ---------------------------------------------------------------------- 2022-08-17T13:43:11.5098390Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s) 2022-08-17T13:43:11.5098703Z 2022-08-17T13:43:11.5098982Z ---------------------------------------------------------------------- 2022-08-17T13:43:11.5099254Z Ran 1 test in 0.003s 2022-08-17T13:43:11.5099377Z 2022-08-17T13:43:11.5099440Z OK (skipped=1) 2022-08-17T13:43:11.5099553Z 2022-08-17T13:43:11.5099641Z Generating XML reports... 2022-08-17T13:43:11.5124268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134311.xml 2022-08-17T13:43:12.6799868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:12.6800240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:12.6802657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:12.6803054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:13.0252955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:13.0262161Z 2022-08-17T13:43:13.0262819Z Running tests... 2022-08-17T13:43:13.0263277Z ---------------------------------------------------------------------- 2022-08-17T13:43:13.2942979Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25551 2022-08-17T13:43:13.2971255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25552 2022-08-17T13:43:13.2999847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25553 2022-08-17T13:43:14.2952361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:14.2953059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:14.2956047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:14.2956640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:14.2967478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:14.2967867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:14.2972337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:14.2972987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:14.2973848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:14.2974484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:14.2977885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:14.2978487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:14.6757687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:14.6758238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:14.6758644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:14.6759282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:14.6759837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:14.6760372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:14.6863735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:14.7769071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:14.7769500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:14.8076666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:43:14.8177945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:43:14.8178738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:43:14.8179778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:14.8180358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:14.8180909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:15.2049922Z ok (2.178s) 2022-08-17T13:43:15.2050171Z 2022-08-17T13:43:15.2050893Z ---------------------------------------------------------------------- 2022-08-17T13:43:15.2051314Z Ran 1 test in 2.179s 2022-08-17T13:43:15.2051492Z 2022-08-17T13:43:15.2051591Z OK 2022-08-17T13:43:15.2051751Z 2022-08-17T13:43:15.2051896Z Generating XML reports... 2022-08-17T13:43:15.2086471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134313.xml 2022-08-17T13:43:16.4750351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:16.4750746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:16.4753494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:16.4753896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:16.8197778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:16.8206812Z 2022-08-17T13:43:16.8207008Z Running tests... 2022-08-17T13:43:16.8207413Z ---------------------------------------------------------------------- 2022-08-17T13:43:17.0907444Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25613 2022-08-17T13:43:17.0936317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25614 2022-08-17T13:43:17.0965459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25615 2022-08-17T13:43:18.0718203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:18.0718862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:18.0721433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:18.0722033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:18.0771338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:18.0772012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:18.0775482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:18.0776139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:18.0783447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:18.0784368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:18.0788104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:18.0788751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:18.4470344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:18.4571585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:18.4572307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:18.4572937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:18.4573517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:18.4574079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:18.4579689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:18.4580508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:18.4581417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:43:18.4581808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:18.4690418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:43:18.4690834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:43:18.4691467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:18.4692037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:18.4786135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:19.0015380Z ok (2.181s) 2022-08-17T13:43:19.0015556Z 2022-08-17T13:43:19.0016038Z ---------------------------------------------------------------------- 2022-08-17T13:43:19.0016286Z Ran 1 test in 2.181s 2022-08-17T13:43:19.0016411Z 2022-08-17T13:43:19.0016476Z OK 2022-08-17T13:43:19.0016572Z 2022-08-17T13:43:19.0016666Z Generating XML reports... 2022-08-17T13:43:19.0051971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134316.xml 2022-08-17T13:43:20.3252888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:20.3253299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:20.3255705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:20.3256083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:20.6793151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:20.6802836Z 2022-08-17T13:43:20.6802980Z Running tests... 2022-08-17T13:43:20.6803622Z ---------------------------------------------------------------------- 2022-08-17T13:43:20.9473840Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25672 2022-08-17T13:43:20.9502960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25673 2022-08-17T13:43:20.9531024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25674 2022-08-17T13:43:21.9515505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:21.9516213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:21.9518887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:21.9519303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:21.9603960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:21.9604660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:21.9607739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:21.9608407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:21.9943378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:21.9944108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:21.9947238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:21.9947658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:22.3631061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:22.3732916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:22.3733353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:22.3734005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:22.3734577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:22.3735107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:22.3839633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:22.4746723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:22.4747153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:22.7578378Z skip: Need at least 2 CUDA devices (2.077s) 2022-08-17T13:43:22.7578702Z 2022-08-17T13:43:22.7579222Z ---------------------------------------------------------------------- 2022-08-17T13:43:22.7579697Z Ran 1 test in 2.077s 2022-08-17T13:43:22.7579916Z 2022-08-17T13:43:22.7579990Z OK (skipped=1) 2022-08-17T13:43:22.7580102Z 2022-08-17T13:43:22.7580205Z Generating XML reports... 2022-08-17T13:43:22.7618196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134320.xml 2022-08-17T13:43:24.0481301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:24.0481714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:24.0484080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:24.0484442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:24.3958902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:24.3969124Z 2022-08-17T13:43:24.3969386Z Running tests... 2022-08-17T13:43:24.3969761Z ---------------------------------------------------------------------- 2022-08-17T13:43:24.3991405Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:43:24.3992173Z 2022-08-17T13:43:24.3992528Z ---------------------------------------------------------------------- 2022-08-17T13:43:24.3992788Z Ran 1 test in 0.002s 2022-08-17T13:43:24.3992910Z 2022-08-17T13:43:24.3992983Z OK (skipped=1) 2022-08-17T13:43:24.3993101Z 2022-08-17T13:43:24.3993192Z Generating XML reports... 2022-08-17T13:43:24.4018017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134324.xml 2022-08-17T13:43:25.5750490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:25.5750884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:25.5753611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:25.5754128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:25.9201440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:25.9211331Z 2022-08-17T13:43:25.9211477Z Running tests... 2022-08-17T13:43:25.9212355Z ---------------------------------------------------------------------- 2022-08-17T13:43:26.1891309Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25735 2022-08-17T13:43:26.1919415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25736 2022-08-17T13:43:26.1947723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25737 2022-08-17T13:43:27.1812488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:27.1813098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:27.1816001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:27.1816645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:27.2133051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:27.2133712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:27.2137249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:27.2137925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:27.2445484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:27.2446481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:27.2449647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:27.2450075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:27.5982235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:27.6082770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:27.6083650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:27.6084372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:27.6084946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:27.6085483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:27.6091186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:27.6092980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:27.6093584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:27.8993878Z ok (1.978s) 2022-08-17T13:43:27.8994148Z 2022-08-17T13:43:27.8994647Z ---------------------------------------------------------------------- 2022-08-17T13:43:27.8994925Z Ran 1 test in 1.978s 2022-08-17T13:43:27.8995051Z 2022-08-17T13:43:27.8995115Z OK 2022-08-17T13:43:27.8995212Z 2022-08-17T13:43:27.8995312Z Generating XML reports... 2022-08-17T13:43:27.9029732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134325.xml 2022-08-17T13:43:29.1872855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:29.1873280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:29.1875892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:29.1876510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:29.5322482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:29.5332701Z 2022-08-17T13:43:29.5332806Z Running tests... 2022-08-17T13:43:29.5333276Z ---------------------------------------------------------------------- 2022-08-17T13:43:29.8032758Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25788 2022-08-17T13:43:29.8062208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25789 2022-08-17T13:43:29.8093294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25790 2022-08-17T13:43:30.7910960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:30.7911582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:30.7913828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:30.7914422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:30.8307283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:30.8308001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:30.8310439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:30.8311123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:30.8449619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:30.8450019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:30.8452628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:30.8453154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:31.2039504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:31.2141046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:31.2141720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:31.2142812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:31.2143614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:31.2144425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:31.2150471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:31.2151008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:31.2151762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:31.2154312Z STAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2155036Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2156202Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2162383Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2162780Z 2022-08-17T13:43:31.2163181Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2168683Z STAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2169379Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2173593Z STAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2177350Z STAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2178048Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2181387Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2182401Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2185807Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2189244Z STAGE:2022-08-17 13:43:31 25788:25788 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2197655Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:31.2201325Z STAGE:2022-08-17 13:43:31 25789:25789 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.2201896Z STAGE:2022-08-17 13:43:31 25790:25790 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:31.7142998Z ok (2.181s) 2022-08-17T13:43:31.7143246Z 2022-08-17T13:43:31.7143757Z ---------------------------------------------------------------------- 2022-08-17T13:43:31.7144232Z Ran 1 test in 2.181s 2022-08-17T13:43:31.7144420Z 2022-08-17T13:43:31.7144514Z OK 2022-08-17T13:43:31.7144637Z 2022-08-17T13:43:31.7144787Z Generating XML reports... 2022-08-17T13:43:31.7179737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134329.xml 2022-08-17T13:43:32.9983245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:32.9983652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:32.9985796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:32.9986357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:33.3348915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:33.3358697Z 2022-08-17T13:43:33.3358851Z Running tests... 2022-08-17T13:43:33.3359490Z ---------------------------------------------------------------------- 2022-08-17T13:43:33.6013808Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25844 2022-08-17T13:43:33.6040661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25845 2022-08-17T13:43:33.6069543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25846 2022-08-17T13:43:34.6180716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:34.6181388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:34.6184516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:34.6185115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:34.6528899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:34.6529528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:34.6533085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:34.6533591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:34.6794692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:34.6795329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:34.6797982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:34.6798630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:35.0627661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:35.0628098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:35.0628563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:35.0629269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:35.0629829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:35.0630442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:35.0733830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:35.1640153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:35.1640588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:35.5119997Z ok (2.176s) 2022-08-17T13:43:35.5120241Z 2022-08-17T13:43:35.5120757Z ---------------------------------------------------------------------- 2022-08-17T13:43:35.5121237Z Ran 1 test in 2.176s 2022-08-17T13:43:35.5121361Z 2022-08-17T13:43:35.5121426Z OK 2022-08-17T13:43:35.5121525Z 2022-08-17T13:43:35.5121609Z Generating XML reports... 2022-08-17T13:43:35.5156220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134333.xml 2022-08-17T13:43:36.8148891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:36.8149286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:36.8151310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:36.8151962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:37.1561864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:37.1571151Z 2022-08-17T13:43:37.1571287Z Running tests... 2022-08-17T13:43:37.1571959Z ---------------------------------------------------------------------- 2022-08-17T13:43:37.1589272Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:43:37.1589620Z 2022-08-17T13:43:37.1590003Z ---------------------------------------------------------------------- 2022-08-17T13:43:37.1590447Z Ran 1 test in 0.002s 2022-08-17T13:43:37.1590647Z 2022-08-17T13:43:37.1590731Z OK (skipped=1) 2022-08-17T13:43:37.1590843Z 2022-08-17T13:43:37.1590935Z Generating XML reports... 2022-08-17T13:43:37.1617239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134337.xml 2022-08-17T13:43:38.3072388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:38.3072778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:38.3075664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:38.3076036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:38.6454318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:38.6463831Z 2022-08-17T13:43:38.6463974Z Running tests... 2022-08-17T13:43:38.6464582Z ---------------------------------------------------------------------- 2022-08-17T13:43:38.9081976Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25907 2022-08-17T13:43:38.9110516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25908 2022-08-17T13:43:38.9139246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25909 2022-08-17T13:43:39.9118899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:39.9119330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:39.9122229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:39.9122678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:39.9149882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:39.9150249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:39.9154198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:39.9154669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:39.9190412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:39.9190779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:39.9194523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:39.9194947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:40.2942721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:40.3033576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:40.3034008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:40.3034864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:40.3035428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:40.3043854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:40.3142245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:40.3142747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:40.4055434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:40.4356026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:43:40.4356640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:43:40.4357275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:43:40.4358293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:40.4359323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:40.4360204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:40.4361525Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4362122Z 2022-08-17T13:43:40.4362136Z 2022-08-17T13:43:40.4367827Z STAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4368620Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4369392Z STAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4373627Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4374129Z STAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4378094Z STAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4381016Z STAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4384683Z STAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4388763Z STAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4389908Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4391989Z STAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4395054Z STAGE:2022-08-17 13:43:40 25907:25907 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4402078Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:40.4406337Z STAGE:2022-08-17 13:43:40 25908:25908 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.4406797Z STAGE:2022-08-17 13:43:40 25909:25909 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:40.9191534Z ok (2.272s) 2022-08-17T13:43:40.9191760Z 2022-08-17T13:43:40.9192560Z ---------------------------------------------------------------------- 2022-08-17T13:43:40.9192937Z Ran 1 test in 2.273s 2022-08-17T13:43:40.9193060Z 2022-08-17T13:43:40.9193112Z OK 2022-08-17T13:43:40.9193209Z 2022-08-17T13:43:40.9193303Z Generating XML reports... 2022-08-17T13:43:40.9227748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134338.xml 2022-08-17T13:43:42.2320004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:42.2320393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:42.2322945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:42.2323325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:42.5731159Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:42.5741177Z 2022-08-17T13:43:42.5741323Z Running tests... 2022-08-17T13:43:42.8345467Z ---------------------------------------------------------------------- 2022-08-17T13:43:42.8346175Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25972 2022-08-17T13:43:42.8374687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25973 2022-08-17T13:43:42.8403471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25974 2022-08-17T13:43:43.8493289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:43.8493990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:43.8494709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:43.8495203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:43.8496178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:43.8496842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:43.8497900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:43.8498414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:43.8710158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:43.8710541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:43.8713414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:43.8713908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:44.2319143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:44.2326750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:44.2327406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:44.2328146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:44.2328803Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:44.2420554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:44.2436256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:44.2436721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:44.2543464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:43:44.2543996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:43:44.3432187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:44.3434705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:43:44.3435382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:44.3454404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:44.3454949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:44.3457706Z STAGE:2022-08-17 13:43:44 25973:25973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:44.3458178Z STAGE:2022-08-17 13:43:44 25974:25974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:44.3464415Z STAGE:2022-08-17 13:43:44 25974:25974 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:44.3465117Z STAGE:2022-08-17 13:43:44 25973:25973 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:44.3469708Z STAGE:2022-08-17 13:43:44 25974:25974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:44.3479427Z STAGE:2022-08-17 13:43:44 25973:25973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:43:44.3483193Z STAGE:2022-08-17 13:43:44 25973:25973 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:44.3483706Z STAGE:2022-08-17 13:43:44 25974:25974 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:43:44.8454890Z ok (2.271s) 2022-08-17T13:43:44.8455177Z 2022-08-17T13:43:44.8455699Z ---------------------------------------------------------------------- 2022-08-17T13:43:44.8455974Z Ran 1 test in 2.271s 2022-08-17T13:43:44.8456095Z 2022-08-17T13:43:44.8456159Z OK 2022-08-17T13:43:44.8456241Z 2022-08-17T13:43:44.8456350Z Generating XML reports... 2022-08-17T13:43:44.8493748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134342.xml 2022-08-17T13:43:46.1651674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:46.1652321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:46.1654908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:46.1655502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:46.5165813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:46.5176415Z 2022-08-17T13:43:46.5176896Z Running tests... 2022-08-17T13:43:46.5177326Z ---------------------------------------------------------------------- 2022-08-17T13:43:46.7858680Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26033 2022-08-17T13:43:46.7886964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26034 2022-08-17T13:43:46.7914251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26035 2022-08-17T13:43:47.7696792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:47.7697411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:47.7699638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:47.7700604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:47.7790095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:47.7790765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:47.7793968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:47.7794626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:47.8075086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:47.8075729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:47.8078629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:47.8079277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:48.1654042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:48.1706680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:48.1707681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:48.1708352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:48.1708895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:48.1755749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:48.1815367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:48.1816112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:48.2768784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:48.5964985Z ok (2.079s) 2022-08-17T13:43:48.5965257Z 2022-08-17T13:43:48.5965774Z ---------------------------------------------------------------------- 2022-08-17T13:43:48.5966391Z Ran 1 test in 2.079s 2022-08-17T13:43:48.5966576Z 2022-08-17T13:43:48.5966670Z OK 2022-08-17T13:43:48.5966803Z 2022-08-17T13:43:48.5966947Z Generating XML reports... 2022-08-17T13:43:48.6002826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134346.xml 2022-08-17T13:43:49.9083471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:49.9083860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:49.9087155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:49.9087560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:50.2544830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:50.2554790Z 2022-08-17T13:43:50.2555176Z Running tests... 2022-08-17T13:43:50.2555858Z ---------------------------------------------------------------------- 2022-08-17T13:43:50.5247865Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.269s) 2022-08-17T13:43:50.5248395Z 2022-08-17T13:43:50.5248808Z ---------------------------------------------------------------------- 2022-08-17T13:43:50.5249086Z Ran 1 test in 0.269s 2022-08-17T13:43:50.5249193Z 2022-08-17T13:43:50.5249272Z OK (skipped=1) 2022-08-17T13:43:50.5249384Z 2022-08-17T13:43:50.5249473Z Generating XML reports... 2022-08-17T13:43:50.5271391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134350.xml 2022-08-17T13:43:51.7510165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:51.7510895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:51.7513983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:51.7514602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:52.0917732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:52.0928479Z 2022-08-17T13:43:52.0928906Z Running tests... 2022-08-17T13:43:52.0929337Z ---------------------------------------------------------------------- 2022-08-17T13:43:52.3548793Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26096 2022-08-17T13:43:52.3577470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26097 2022-08-17T13:43:52.3605798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26098 2022-08-17T13:43:53.4092928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:53.4093328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:53.4095673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:53.4096242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:53.4398882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:53.4399566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:53.4403353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:53.4403969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:53.4531201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:53.4531864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:53.4535523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:53.4536177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:53.8051443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:53.8151856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:53.8152550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:53.8153232Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:53.8153778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:53.8154337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:53.8258774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:53.9163807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:53.9164608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:53.9172940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:43:53.9269084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:43:53.9271737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:43:53.9272412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:53.9370358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:53.9371516Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:43:54.3658440Z ok (2.273s) 2022-08-17T13:43:54.3658630Z 2022-08-17T13:43:54.3659179Z ---------------------------------------------------------------------- 2022-08-17T13:43:54.3659457Z Ran 1 test in 2.273s 2022-08-17T13:43:54.3659582Z 2022-08-17T13:43:54.3659647Z OK 2022-08-17T13:43:54.3659743Z 2022-08-17T13:43:54.3659986Z Generating XML reports... 2022-08-17T13:43:54.3698202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134352.xml 2022-08-17T13:43:55.6419319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:55.6419730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:55.6422503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:55.6422878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:55.9807760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:55.9817749Z 2022-08-17T13:43:55.9817840Z Running tests... 2022-08-17T13:43:55.9818295Z ---------------------------------------------------------------------- 2022-08-17T13:43:56.2444101Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26155 2022-08-17T13:43:56.2473188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26156 2022-08-17T13:43:56.2500404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26157 2022-08-17T13:43:57.2904166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:57.2904811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:57.2907558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:57.2908267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:57.3261819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:57.3262590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:57.3265157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:57.3265741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:57.3268999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:57.3269655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:57.3273895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:57.3274899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:57.6964860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:43:57.6965567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:43:57.6966396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:43:57.6967338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:57.6968144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:57.6968955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:43:57.7070480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:43:57.7978042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:43:57.7978503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:43:58.1552250Z ok (2.173s) 2022-08-17T13:43:58.1552511Z 2022-08-17T13:43:58.1553247Z ---------------------------------------------------------------------- 2022-08-17T13:43:58.1553652Z Ran 1 test in 2.173s 2022-08-17T13:43:58.1553847Z 2022-08-17T13:43:58.1553946Z OK 2022-08-17T13:43:58.1554099Z 2022-08-17T13:43:58.1554242Z Generating XML reports... 2022-08-17T13:43:58.1589533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134355.xml 2022-08-17T13:43:59.4497671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:43:59.4498059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:43:59.4500705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:43:59.4501102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:43:59.8011688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:43:59.8020935Z 2022-08-17T13:43:59.8021102Z Running tests... 2022-08-17T13:43:59.8021453Z ---------------------------------------------------------------------- 2022-08-17T13:44:00.0677424Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26208 2022-08-17T13:44:00.0708339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26209 2022-08-17T13:44:00.0735951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26210 2022-08-17T13:44:01.0579188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:01.0579599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:01.0582870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:01.0583378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:01.0684637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:01.0685150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:01.0688645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:01.0689296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:01.0752631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:01.0753565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:01.0756085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:01.0756750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:01.4423088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:01.4524340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:01.4524919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:01.4525578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:01.4526308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:01.4526877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:01.4532627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:01.4533930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:01.4534377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:01.9787308Z ok (2.176s) 2022-08-17T13:44:01.9787586Z 2022-08-17T13:44:01.9787998Z ---------------------------------------------------------------------- 2022-08-17T13:44:01.9788285Z Ran 1 test in 2.176s 2022-08-17T13:44:01.9788407Z 2022-08-17T13:44:01.9788473Z OK 2022-08-17T13:44:01.9788573Z 2022-08-17T13:44:01.9788671Z Generating XML reports... 2022-08-17T13:44:01.9826303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134359.xml 2022-08-17T13:44:03.2996169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:03.2996559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:03.2998806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:03.2999218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:03.6426161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:03.6435754Z 2022-08-17T13:44:03.6435919Z Running tests... 2022-08-17T13:44:03.6436297Z ---------------------------------------------------------------------- 2022-08-17T13:44:03.9136924Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26261 2022-08-17T13:44:03.9164353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26262 2022-08-17T13:44:03.9192409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26263 2022-08-17T13:44:04.9008859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:04.9009298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:04.9018937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:04.9019295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:04.9024150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:04.9024782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:04.9037425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:04.9038380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:04.9200552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:04.9200966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:04.9204303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:04.9204692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:05.2857301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:05.2958902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:05.2959583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:05.2960487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:05.2961040Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:05.2962137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:05.2967406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:05.2968318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:05.2969208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:05.3177750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:44:05.3280416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:44:05.3281122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:44:05.3282047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:05.3282603Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:05.3283161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:05.7239432Z ok (2.080s) 2022-08-17T13:44:05.7239694Z 2022-08-17T13:44:05.7240181Z ---------------------------------------------------------------------- 2022-08-17T13:44:05.7240595Z Ran 1 test in 2.080s 2022-08-17T13:44:05.7240787Z 2022-08-17T13:44:05.7240868Z OK 2022-08-17T13:44:05.7241187Z 2022-08-17T13:44:05.7241529Z Generating XML reports... 2022-08-17T13:44:05.7276294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134403.xml 2022-08-17T13:44:07.0485428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:07.0486044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:07.0488492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:07.0489044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:07.3943294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:07.3952997Z 2022-08-17T13:44:07.3953140Z Running tests... 2022-08-17T13:44:07.3953761Z ---------------------------------------------------------------------- 2022-08-17T13:44:07.6616970Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26323 2022-08-17T13:44:07.6645116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26324 2022-08-17T13:44:07.6672962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26325 2022-08-17T13:44:08.6701247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:08.6701907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:08.6704787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:08.6705414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:08.6983592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:08.6984225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:08.6985646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:08.6986262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:08.7051930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:08.7052570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:08.7054918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:08.7055580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:09.0728758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:09.0829409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:09.0830182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:09.0830880Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:09.0831447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:09.0831983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:09.0937550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:09.1843687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:09.1844246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:09.1844801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:44:09.1949094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:44:09.1950016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:44:09.1951012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:09.2050305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:09.2052048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:09.5721374Z ok (2.176s) 2022-08-17T13:44:09.5721633Z 2022-08-17T13:44:09.5722099Z ---------------------------------------------------------------------- 2022-08-17T13:44:09.5722385Z Ran 1 test in 2.177s 2022-08-17T13:44:09.5722505Z 2022-08-17T13:44:09.5722569Z OK 2022-08-17T13:44:09.5722883Z 2022-08-17T13:44:09.5722978Z Generating XML reports... 2022-08-17T13:44:09.5759593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134407.xml 2022-08-17T13:44:10.8859265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:10.8859670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:10.8862409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:10.8862936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:11.2331889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:11.2341492Z 2022-08-17T13:44:11.2341626Z Running tests... 2022-08-17T13:44:11.2342297Z ---------------------------------------------------------------------- 2022-08-17T13:44:11.2369516Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s) 2022-08-17T13:44:11.2369970Z 2022-08-17T13:44:11.2370359Z ---------------------------------------------------------------------- 2022-08-17T13:44:11.2370638Z Ran 1 test in 0.003s 2022-08-17T13:44:11.2370759Z 2022-08-17T13:44:11.2371081Z OK (skipped=1) 2022-08-17T13:44:11.2371198Z 2022-08-17T13:44:11.2371275Z Generating XML reports... 2022-08-17T13:44:11.2394988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134411.xml 2022-08-17T13:44:12.3996616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:12.3996999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:12.3999530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:12.3999921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:12.7392075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:12.7402011Z 2022-08-17T13:44:12.7402146Z Running tests... 2022-08-17T13:44:12.7402658Z ---------------------------------------------------------------------- 2022-08-17T13:44:13.0021695Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26392 2022-08-17T13:44:13.0049999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26393 2022-08-17T13:44:13.0078236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26394 2022-08-17T13:44:14.0268508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:14.0269181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:14.0271188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:14.0271834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:14.0344566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:14.0345222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:14.0348577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:14.0349247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:14.0526389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:14.0526985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:14.0529164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:14.0530046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:14.4277178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:14.4379497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:14.4379981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:14.4380787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:14.4381432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:14.4382388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:14.4389278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:14.4389980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:14.4390897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:14.9127651Z ok (2.172s) 2022-08-17T13:44:14.9127889Z 2022-08-17T13:44:14.9128418Z ---------------------------------------------------------------------- 2022-08-17T13:44:14.9128701Z Ran 1 test in 2.172s 2022-08-17T13:44:14.9128826Z 2022-08-17T13:44:14.9128893Z OK 2022-08-17T13:44:14.9128990Z 2022-08-17T13:44:14.9129071Z Generating XML reports... 2022-08-17T13:44:14.9166780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134412.xml 2022-08-17T13:44:16.2037790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:16.2038204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:16.2040652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:16.2041040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:16.5548558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:16.5558280Z 2022-08-17T13:44:16.5558740Z Running tests... 2022-08-17T13:44:16.5559182Z ---------------------------------------------------------------------- 2022-08-17T13:44:16.8159965Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26445 2022-08-17T13:44:16.8187430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26446 2022-08-17T13:44:16.8216158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26447 2022-08-17T13:44:17.8144525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:17.8145259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:17.8147660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:17.8148278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:17.8250157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:17.8250877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:17.8254060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:17.8254441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:17.8518553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:17.8519239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:17.8522189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:17.8522867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:18.2138559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:18.2239707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:18.2240275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:18.2240933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:18.2241504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:18.2242062Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:18.2349012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:18.3254020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:18.3254687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:18.7268371Z ok (2.171s) 2022-08-17T13:44:18.7268582Z 2022-08-17T13:44:18.7268898Z ---------------------------------------------------------------------- 2022-08-17T13:44:18.7269175Z Ran 1 test in 2.171s 2022-08-17T13:44:18.7269296Z 2022-08-17T13:44:18.7269361Z OK 2022-08-17T13:44:18.7269455Z 2022-08-17T13:44:18.7269538Z Generating XML reports... 2022-08-17T13:44:18.7307476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134416.xml 2022-08-17T13:44:20.0332524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:20.0333143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:20.0335212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:20.0335829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:20.3749769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:20.3759594Z 2022-08-17T13:44:20.3760022Z Running tests... 2022-08-17T13:44:20.3760470Z ---------------------------------------------------------------------- 2022-08-17T13:44:20.6444617Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26498 2022-08-17T13:44:20.6475511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26499 2022-08-17T13:44:20.6503746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26500 2022-08-17T13:44:21.6367597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:21.6368077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:21.6370831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:21.6371198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:21.6575310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:21.6575677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:21.6579221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:21.6579593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:21.6669867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:21.6670228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:21.6674131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:21.6674519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:22.0388800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:22.0490577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:22.0491275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:22.0492247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:22.0493416Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:22.0493989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:22.0596983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:22.0599328Z STAGE:2022-08-17 13:44:22 26500:26500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:22.1503476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:22.1504085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:22.1505031Z STAGE:2022-08-17 13:44:22 26498:26498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:22.1506737Z STAGE:2022-08-17 13:44:22 26499:26499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:22.1537634Z STAGE:2022-08-17 13:44:22 26498:26498 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:22.1540152Z STAGE:2022-08-17 13:44:22 26499:26499 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:22.1541330Z STAGE:2022-08-17 13:44:22 26500:26500 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:22.5554688Z ok (2.179s) 2022-08-17T13:44:22.5554939Z 2022-08-17T13:44:22.5555492Z ---------------------------------------------------------------------- 2022-08-17T13:44:22.5555810Z Ran 1 test in 2.179s 2022-08-17T13:44:22.5555932Z 2022-08-17T13:44:22.5555983Z OK 2022-08-17T13:44:22.5556080Z 2022-08-17T13:44:22.5556176Z Generating XML reports... 2022-08-17T13:44:22.5593434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134420.xml 2022-08-17T13:44:23.8398937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:23.8399347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:23.8401522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:23.8401905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:24.1855726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:24.1865509Z 2022-08-17T13:44:24.1865629Z Running tests... 2022-08-17T13:44:24.1866105Z ---------------------------------------------------------------------- 2022-08-17T13:44:24.4563321Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26554 2022-08-17T13:44:24.4592662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26555 2022-08-17T13:44:24.4620963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26556 2022-08-17T13:44:25.4524704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:25.4525381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:25.4527863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:25.4528516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:25.4575074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:25.4575730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:25.4578883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:25.4579517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:25.4831740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:25.4832454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:25.4835350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:25.4835951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:25.8482249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:25.8582886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:25.8583501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:25.8584157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:25.8584731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:25.8585267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:25.8690092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:25.8692896Z STAGE:2022-08-17 13:44:25 26556:26556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:25.9595525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:25.9596002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:25.9598477Z STAGE:2022-08-17 13:44:25 26555:26555 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:25.9599316Z STAGE:2022-08-17 13:44:25 26554:26554 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:44:25.9630001Z STAGE:2022-08-17 13:44:25 26555:26555 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:25.9630653Z STAGE:2022-08-17 13:44:25 26556:26556 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:26.0626288Z STAGE:2022-08-17 13:44:26 26554:26554 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:44:26.3671203Z ok (2.180s) 2022-08-17T13:44:26.3671408Z 2022-08-17T13:44:26.3671825Z ---------------------------------------------------------------------- 2022-08-17T13:44:26.3672113Z Ran 1 test in 2.181s 2022-08-17T13:44:26.3672235Z 2022-08-17T13:44:26.3672306Z OK 2022-08-17T13:44:26.3672401Z 2022-08-17T13:44:26.3672709Z Generating XML reports... 2022-08-17T13:44:26.3708086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134424.xml 2022-08-17T13:44:27.6619466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:27.6619889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:27.6622329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:27.6622708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:28.0046761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:28.0056911Z 2022-08-17T13:44:28.0057379Z Running tests... 2022-08-17T13:44:28.0057948Z ---------------------------------------------------------------------- 2022-08-17T13:44:28.0076311Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:44:28.0076809Z 2022-08-17T13:44:28.0077196Z ---------------------------------------------------------------------- 2022-08-17T13:44:28.0077466Z Ran 1 test in 0.002s 2022-08-17T13:44:28.0077585Z 2022-08-17T13:44:28.0077872Z OK (skipped=1) 2022-08-17T13:44:28.0077995Z 2022-08-17T13:44:28.0078085Z Generating XML reports... 2022-08-17T13:44:28.0101637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134428.xml 2022-08-17T13:44:29.1947344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:29.1947721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:29.1950192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:29.1950706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:29.5428337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:29.5438882Z 2022-08-17T13:44:29.5439097Z Running tests... 2022-08-17T13:44:29.5439751Z ---------------------------------------------------------------------- 2022-08-17T13:44:29.5458110Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:44:29.5458639Z 2022-08-17T13:44:29.5458916Z ---------------------------------------------------------------------- 2022-08-17T13:44:29.5459174Z Ran 1 test in 0.002s 2022-08-17T13:44:29.5459295Z 2022-08-17T13:44:29.5459372Z OK (skipped=1) 2022-08-17T13:44:29.5459485Z 2022-08-17T13:44:29.5459575Z Generating XML reports... 2022-08-17T13:44:29.5486668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134429.xml 2022-08-17T13:44:30.7144192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:30.7144589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:30.7147771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:30.7148171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:31.0525747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:31.0535466Z 2022-08-17T13:44:31.0535654Z Running tests... 2022-08-17T13:44:31.0536021Z ---------------------------------------------------------------------- 2022-08-17T13:44:31.3143366Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26630 2022-08-17T13:44:31.3172074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26631 2022-08-17T13:44:31.3199885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26632 2022-08-17T13:44:32.3233449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:32.3234152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:32.3235271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:32.3235896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:32.3350230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:32.3350911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:32.3355290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:32.3355976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:32.3358520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:32.3359152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:32.3363256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:32.3363876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:32.6992268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:32.7092697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:32.7093272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:32.7093940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:32.7094501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:32.7095045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:32.7201127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:32.7201685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:32.8105720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:34.8131099Z [E ProcessGroupGloo.cpp:136] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:44:34.8229436Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:44:35.2470157Z ok (4.193s) 2022-08-17T13:44:35.2470597Z 2022-08-17T13:44:35.2471012Z ---------------------------------------------------------------------- 2022-08-17T13:44:35.2471458Z Ran 1 test in 4.193s 2022-08-17T13:44:35.2471613Z 2022-08-17T13:44:35.2471679Z OK 2022-08-17T13:44:35.2471780Z 2022-08-17T13:44:35.2471879Z Generating XML reports... 2022-08-17T13:44:35.2506536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134431.xml 2022-08-17T13:44:36.5213346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:36.5213733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:36.5216531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:36.5217249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:36.8605818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:36.8615653Z 2022-08-17T13:44:36.8615915Z Running tests... 2022-08-17T13:44:37.1257370Z ---------------------------------------------------------------------- 2022-08-17T13:44:37.1257879Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26683 2022-08-17T13:44:37.1284618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26684 2022-08-17T13:44:37.1312806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26685 2022-08-17T13:44:38.0898190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:38.0898606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:38.0901371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:38.0901896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:38.1022913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:38.1023265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:38.1027043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:38.1027414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:38.1284188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:38.1284534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:38.1288724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:38.1289358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:38.4851451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:38.4952277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:38.4952983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:38.4953624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:38.4954178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:38.4954724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:38.5059616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:38.5964095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:38.5964737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:40.6166991Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:44:40.6167871Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:44:41.0389866Z ok (4.177s) 2022-08-17T13:44:41.0391096Z 2022-08-17T13:44:41.0391954Z ---------------------------------------------------------------------- 2022-08-17T13:44:41.0392620Z Ran 1 test in 4.177s 2022-08-17T13:44:41.0393103Z 2022-08-17T13:44:41.0394338Z OK 2022-08-17T13:44:41.0394952Z 2022-08-17T13:44:41.0395094Z Generating XML reports... 2022-08-17T13:44:41.0429045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134436.xml 2022-08-17T13:44:42.3367815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:42.3368212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:42.3370551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:42.3370983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:42.6766731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:42.6776514Z 2022-08-17T13:44:42.6776615Z Running tests... 2022-08-17T13:44:42.6777224Z ---------------------------------------------------------------------- 2022-08-17T13:44:42.9397703Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26736 2022-08-17T13:44:42.9425662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26737 2022-08-17T13:44:42.9454212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26738 2022-08-17T13:44:43.9186895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:43.9187526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:43.9188299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:43.9188879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:43.9189620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:43.9190228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:43.9190968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:43.9191606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:43.9192403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:43.9193001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:43.9193797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:43.9194427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:44.3037575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:44.3138014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:44.3138608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:44.3139276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:44.3139833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:44.3140388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:44.3247563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:44.3248112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:44.4151072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:44.4259296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:44:44.4362085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:44:44.4362676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:44:44.4363627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:44.4364584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:44.4365578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:44.4366531Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-08-17T13:44:44.4367100Z No ranks successfully processed in monitoredBarrier. 2022-08-17T13:44:44.4394292Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-08-17T13:44:44.8504373Z ok (2.172s) 2022-08-17T13:44:44.8504610Z 2022-08-17T13:44:44.8505171Z ---------------------------------------------------------------------- 2022-08-17T13:44:44.8505533Z Ran 1 test in 2.173s 2022-08-17T13:44:44.8505868Z 2022-08-17T13:44:44.8505942Z OK 2022-08-17T13:44:44.8506039Z 2022-08-17T13:44:44.8506140Z Generating XML reports... 2022-08-17T13:44:44.8540703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134442.xml 2022-08-17T13:44:46.1485356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:46.1485766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:46.1488610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:46.1489147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:46.4929602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:46.4939030Z 2022-08-17T13:44:46.4939176Z Running tests... 2022-08-17T13:44:46.4939821Z ---------------------------------------------------------------------- 2022-08-17T13:44:46.7632452Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26798 2022-08-17T13:44:46.7660560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26799 2022-08-17T13:44:46.7688530Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26800 2022-08-17T13:44:47.7784259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:47.7784923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:47.7785724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:47.7786295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:47.7787084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:47.7787654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:47.7791213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:47.7791881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:47.7805994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:47.7806429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:47.7810524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:47.7810907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:48.1434218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:48.1535433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:48.1536145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:48.1536798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:48.1537360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:48.1537913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:48.1543119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:48.1544069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:48.1545170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:48.1547523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:44:48.1650152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:44:48.1650865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:44:48.1651483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:48.1652193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:48.1751175Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:44:48.1756769Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:299: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group. 2022-08-17T13:44:48.1757221Z f"Running {op_name} on global rank {global_rank} which does not " 2022-08-17T13:44:48.2756501Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:44:48.6739069Z ok (2.180s) 2022-08-17T13:44:48.6739356Z 2022-08-17T13:44:48.6739863Z ---------------------------------------------------------------------- 2022-08-17T13:44:48.6740160Z Ran 1 test in 2.180s 2022-08-17T13:44:48.6740282Z 2022-08-17T13:44:48.6740346Z OK 2022-08-17T13:44:48.6740442Z 2022-08-17T13:44:48.6740541Z Generating XML reports... 2022-08-17T13:44:48.6775740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134446.xml 2022-08-17T13:44:49.9961392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:49.9961815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:49.9964239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:49.9964723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:50.3416695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:50.3426849Z 2022-08-17T13:44:50.3427035Z Running tests... 2022-08-17T13:44:50.3427651Z ---------------------------------------------------------------------- 2022-08-17T13:44:50.6217593Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26857 2022-08-17T13:44:50.6246224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26858 2022-08-17T13:44:50.6274269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26859 2022-08-17T13:44:51.6071437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:51.6071828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:51.6075153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:51.6075820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:51.6240392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:51.6241066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:51.6244785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:51.6245468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:51.6502192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:51.6502908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:51.6505363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:51.6506026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:52.0147771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:44:52.0249954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:44:52.0250652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:44:52.0251451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:52.0252049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:52.0252606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:44:52.0259281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:44:52.0259797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:44:52.0261111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:44:52.1453594Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:44:52.1454234Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:44:52.1454592Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms 2022-08-17T13:44:52.5322348Z ok (2.189s) 2022-08-17T13:44:52.5322590Z 2022-08-17T13:44:52.5323165Z ---------------------------------------------------------------------- 2022-08-17T13:44:52.5323553Z Ran 1 test in 2.189s 2022-08-17T13:44:52.5323661Z 2022-08-17T13:44:52.5323725Z OK 2022-08-17T13:44:52.5323821Z 2022-08-17T13:44:52.5323919Z Generating XML reports... 2022-08-17T13:44:52.5361211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134450.xml 2022-08-17T13:44:53.8251442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:53.8252111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:53.8254928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:53.8255503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:54.1679560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:54.1689677Z 2022-08-17T13:44:54.1690015Z Running tests... 2022-08-17T13:44:54.1690644Z ---------------------------------------------------------------------- 2022-08-17T13:44:54.1713971Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:44:54.1714390Z 2022-08-17T13:44:54.1714797Z ---------------------------------------------------------------------- 2022-08-17T13:44:54.1715201Z Ran 1 test in 0.002s 2022-08-17T13:44:54.1715311Z 2022-08-17T13:44:54.1715392Z OK (skipped=1) 2022-08-17T13:44:54.1715506Z 2022-08-17T13:44:54.1715608Z Generating XML reports... 2022-08-17T13:44:54.1740602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134454.xml 2022-08-17T13:44:55.3276818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:55.3277546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:55.3279828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:55.3280206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:55.6736932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:55.6747314Z 2022-08-17T13:44:55.6747443Z Running tests... 2022-08-17T13:44:55.6747925Z ---------------------------------------------------------------------- 2022-08-17T13:44:55.6772917Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:44:55.6773353Z 2022-08-17T13:44:55.6773747Z ---------------------------------------------------------------------- 2022-08-17T13:44:55.6774180Z Ran 1 test in 0.003s 2022-08-17T13:44:55.6774395Z 2022-08-17T13:44:55.6774528Z OK (skipped=1) 2022-08-17T13:44:55.6774736Z 2022-08-17T13:44:55.6774898Z Generating XML reports... 2022-08-17T13:44:55.6799602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134455.xml 2022-08-17T13:44:56.8404980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:56.8405371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:56.8407857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:56.8408375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:57.1842690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:57.1852740Z 2022-08-17T13:44:57.1853060Z Running tests... 2022-08-17T13:44:57.1853736Z ---------------------------------------------------------------------- 2022-08-17T13:44:57.1875716Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-08-17T13:44:57.1876204Z 2022-08-17T13:44:57.1876587Z ---------------------------------------------------------------------- 2022-08-17T13:44:57.1876864Z Ran 1 test in 0.002s 2022-08-17T13:44:57.1876988Z 2022-08-17T13:44:57.1877068Z OK (skipped=1) 2022-08-17T13:44:57.1877184Z 2022-08-17T13:44:57.1877274Z Generating XML reports... 2022-08-17T13:44:57.1901534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134457.xml 2022-08-17T13:44:58.3409629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:58.3410030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:58.3412249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:58.3412626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:44:58.6824799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:44:58.6834474Z 2022-08-17T13:44:58.6834577Z Running tests... 2022-08-17T13:44:58.6835178Z ---------------------------------------------------------------------- 2022-08-17T13:44:58.6862081Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2022-08-17T13:44:58.6862505Z 2022-08-17T13:44:58.6862831Z ---------------------------------------------------------------------- 2022-08-17T13:44:58.6863151Z Ran 1 test in 0.003s 2022-08-17T13:44:58.6863271Z 2022-08-17T13:44:58.6863348Z OK (skipped=1) 2022-08-17T13:44:58.6863463Z 2022-08-17T13:44:58.6863618Z Generating XML reports... 2022-08-17T13:44:58.6888577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134458.xml 2022-08-17T13:44:59.8517000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:44:59.8517407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:44:59.8520027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:44:59.8520398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:00.1969994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:00.1979740Z 2022-08-17T13:45:00.1979879Z Running tests... 2022-08-17T13:45:00.1980272Z ---------------------------------------------------------------------- 2022-08-17T13:45:00.2003494Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-08-17T13:45:00.2004003Z 2022-08-17T13:45:00.2004290Z ---------------------------------------------------------------------- 2022-08-17T13:45:00.2004537Z Ran 1 test in 0.002s 2022-08-17T13:45:00.2004660Z 2022-08-17T13:45:00.2004737Z OK (skipped=1) 2022-08-17T13:45:00.2004854Z 2022-08-17T13:45:00.2004945Z Generating XML reports... 2022-08-17T13:45:00.2029865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134500.xml 2022-08-17T13:45:01.3770323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:01.3770745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:01.3773725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:01.3774299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:01.7280298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:01.7289961Z 2022-08-17T13:45:01.7290613Z Running tests... 2022-08-17T13:45:01.7291075Z ---------------------------------------------------------------------- 2022-08-17T13:45:01.7313249Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:45:01.7313658Z 2022-08-17T13:45:01.7314098Z ---------------------------------------------------------------------- 2022-08-17T13:45:01.7314391Z Ran 1 test in 0.002s 2022-08-17T13:45:01.7314500Z 2022-08-17T13:45:01.7314578Z OK (skipped=1) 2022-08-17T13:45:01.7314690Z 2022-08-17T13:45:01.7315027Z Generating XML reports... 2022-08-17T13:45:01.7339366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134501.xml 2022-08-17T13:45:02.8945400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:02.8945815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:02.8948212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:02.8948572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:03.2394202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:03.2404120Z 2022-08-17T13:45:03.2404213Z Running tests... 2022-08-17T13:45:03.2404810Z ---------------------------------------------------------------------- 2022-08-17T13:45:03.2427974Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:45:03.2428328Z 2022-08-17T13:45:03.2428648Z ---------------------------------------------------------------------- 2022-08-17T13:45:03.2428917Z Ran 1 test in 0.002s 2022-08-17T13:45:03.2429037Z 2022-08-17T13:45:03.2429113Z OK (skipped=1) 2022-08-17T13:45:03.2429466Z 2022-08-17T13:45:03.2429593Z Generating XML reports... 2022-08-17T13:45:03.2454096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134503.xml 2022-08-17T13:45:04.3939082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:04.3939475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:04.3941360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:04.3941742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:04.7301180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:04.7311257Z 2022-08-17T13:45:04.7311353Z Running tests... 2022-08-17T13:45:04.7311815Z ---------------------------------------------------------------------- 2022-08-17T13:45:04.7332522Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:45:04.7332879Z 2022-08-17T13:45:04.7333163Z ---------------------------------------------------------------------- 2022-08-17T13:45:04.7333469Z Ran 1 test in 0.002s 2022-08-17T13:45:04.7333589Z 2022-08-17T13:45:04.7333670Z OK (skipped=1) 2022-08-17T13:45:04.7333783Z 2022-08-17T13:45:04.7333873Z Generating XML reports... 2022-08-17T13:45:04.7358347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134504.xml 2022-08-17T13:45:05.8889602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:05.8890023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:05.8892301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:05.8892680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:06.2331301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:06.2341146Z 2022-08-17T13:45:06.2341233Z Running tests... 2022-08-17T13:45:06.2341719Z ---------------------------------------------------------------------- 2022-08-17T13:45:06.4996649Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26990 2022-08-17T13:45:06.5024349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26991 2022-08-17T13:45:06.5052902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26992 2022-08-17T13:45:07.4771987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:07.4772396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:07.4774637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:07.4775007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:07.4973186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:07.4973553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:07.4977626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:07.4978007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:07.5121071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:07.5121671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:07.5125658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:07.5126188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:07.8735595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:07.8736042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:07.8736950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:07.8737702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:07.8738518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:07.8739338Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:07.8842690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:07.9749250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:07.9749847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:08.3101249Z skip: CUDA is not available. (2.076s) 2022-08-17T13:45:08.3101530Z 2022-08-17T13:45:08.3102074Z ---------------------------------------------------------------------- 2022-08-17T13:45:08.3102454Z Ran 1 test in 2.076s 2022-08-17T13:45:08.3102578Z 2022-08-17T13:45:08.3102656Z OK (skipped=1) 2022-08-17T13:45:08.3102772Z 2022-08-17T13:45:08.3102849Z Generating XML reports... 2022-08-17T13:45:08.3137638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134506.xml 2022-08-17T13:45:09.6277237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:09.6277655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:09.6279967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:09.6280339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:09.9709848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:09.9719533Z 2022-08-17T13:45:09.9719679Z Running tests... 2022-08-17T13:45:09.9720277Z ---------------------------------------------------------------------- 2022-08-17T13:45:10.2431556Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27043 2022-08-17T13:45:10.2459733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27044 2022-08-17T13:45:10.2487951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27045 2022-08-17T13:45:11.2094049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:11.2094484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:11.2097000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:11.2097527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:11.2104687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:11.2105452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:11.2109996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:11.2110632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:11.2643418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:11.2644015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:11.2647553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:11.2648148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:11.6287171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:11.6288110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:11.6288545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:11.6288951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:11.6289469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:11.6290019Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:11.6391612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:11.7299908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:11.7300404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:12.0539508Z skip: CUDA is not available. (2.082s) 2022-08-17T13:45:12.0539777Z 2022-08-17T13:45:12.0540271Z ---------------------------------------------------------------------- 2022-08-17T13:45:12.0540757Z Ran 1 test in 2.082s 2022-08-17T13:45:12.0540880Z 2022-08-17T13:45:12.0540957Z OK (skipped=1) 2022-08-17T13:45:12.0541056Z 2022-08-17T13:45:12.0541149Z Generating XML reports... 2022-08-17T13:45:12.0575836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134509.xml 2022-08-17T13:45:13.3583740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:13.3584462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:13.3587304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:13.3587993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:13.6980937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:13.6990467Z 2022-08-17T13:45:13.6990590Z Running tests... 2022-08-17T13:45:13.6991205Z ---------------------------------------------------------------------- 2022-08-17T13:45:13.7011663Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:45:13.7012151Z 2022-08-17T13:45:13.7012617Z ---------------------------------------------------------------------- 2022-08-17T13:45:13.7013016Z Ran 1 test in 0.002s 2022-08-17T13:45:13.7013140Z 2022-08-17T13:45:13.7013218Z OK (skipped=1) 2022-08-17T13:45:13.7013333Z 2022-08-17T13:45:13.7013428Z Generating XML reports... 2022-08-17T13:45:13.7037438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134513.xml 2022-08-17T13:45:14.8573117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:14.8573514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:14.8576142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:14.8576537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:15.2005290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:15.2015094Z 2022-08-17T13:45:15.2015247Z Running tests... 2022-08-17T13:45:15.2015680Z ---------------------------------------------------------------------- 2022-08-17T13:45:15.2033814Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-08-17T13:45:15.2034796Z 2022-08-17T13:45:15.2035081Z ---------------------------------------------------------------------- 2022-08-17T13:45:15.2035463Z Ran 1 test in 0.002s 2022-08-17T13:45:15.2035588Z 2022-08-17T13:45:15.2035712Z OK (skipped=1) 2022-08-17T13:45:15.2035819Z 2022-08-17T13:45:15.2035910Z Generating XML reports... 2022-08-17T13:45:15.2059998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134515.xml 2022-08-17T13:45:16.3654783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:16.3655206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:16.3658056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:16.3658451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:16.7091168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:16.7100947Z 2022-08-17T13:45:16.7101064Z Running tests... 2022-08-17T13:45:16.7101694Z ---------------------------------------------------------------------- 2022-08-17T13:45:16.9660829Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s) 2022-08-17T13:45:16.9661351Z 2022-08-17T13:45:16.9661568Z ---------------------------------------------------------------------- 2022-08-17T13:45:16.9661841Z Ran 1 test in 0.256s 2022-08-17T13:45:16.9661962Z 2022-08-17T13:45:16.9662024Z OK (skipped=1) 2022-08-17T13:45:16.9662137Z 2022-08-17T13:45:16.9662225Z Generating XML reports... 2022-08-17T13:45:16.9686757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134516.xml 2022-08-17T13:45:18.2183686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:18.2184380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:18.2185939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:18.2186624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:18.5610224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:18.5619756Z 2022-08-17T13:45:18.5619846Z Running tests... 2022-08-17T13:45:18.5620460Z ---------------------------------------------------------------------- 2022-08-17T13:45:18.8261397Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27126 2022-08-17T13:45:18.8288971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27127 2022-08-17T13:45:18.8317020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27128 2022-08-17T13:45:19.8294415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:19.8295060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:19.8297097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:19.8297697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:19.8308853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:19.8309253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:19.8312894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:19.8313432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:19.8394153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:19.8394675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:19.8398590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:19.8399138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:20.2022922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:20.2062379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:20.2063010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:20.2063732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:20.2064309Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:20.2124477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:20.2170816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:20.2171413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:20.3163888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:20.6365178Z skip: Need at least 2 CUDA devices (2.074s) 2022-08-17T13:45:20.6365965Z 2022-08-17T13:45:20.6366371Z ---------------------------------------------------------------------- 2022-08-17T13:45:20.6366657Z Ran 1 test in 2.074s 2022-08-17T13:45:20.6366777Z 2022-08-17T13:45:20.6366853Z OK (skipped=1) 2022-08-17T13:45:20.6366966Z 2022-08-17T13:45:20.6367042Z Generating XML reports... 2022-08-17T13:45:20.6402065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134518.xml 2022-08-17T13:45:21.9381197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:21.9381599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:21.9384176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:21.9384561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:22.2830064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:22.2839688Z 2022-08-17T13:45:22.2839792Z Running tests... 2022-08-17T13:45:22.2840240Z ---------------------------------------------------------------------- 2022-08-17T13:45:22.5575594Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27179 2022-08-17T13:45:22.5602830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27180 2022-08-17T13:45:22.5631602Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27181 2022-08-17T13:45:23.5210118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:23.5210522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:23.5212938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:23.5213363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:23.5371538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:23.5371906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:23.5375550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:23.5376043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:23.5499011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:23.5499654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:23.5502930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:23.5503596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:23.9273978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:23.9274467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:23.9275132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:23.9276016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:23.9276581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:23.9277138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:23.9379521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:24.0286174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:24.0286803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:24.3679731Z skip: Need at least 2 CUDA devices (2.084s) 2022-08-17T13:45:24.3680013Z 2022-08-17T13:45:24.3680511Z ---------------------------------------------------------------------- 2022-08-17T13:45:24.3680860Z Ran 1 test in 2.084s 2022-08-17T13:45:24.3681041Z 2022-08-17T13:45:24.3681481Z OK (skipped=1) 2022-08-17T13:45:24.3681645Z 2022-08-17T13:45:24.3681740Z Generating XML reports... 2022-08-17T13:45:24.3719438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134522.xml 2022-08-17T13:45:25.6815475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:25.6815878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:25.6818672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:25.6819077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:26.0269228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:26.0278848Z 2022-08-17T13:45:26.0278997Z Running tests... 2022-08-17T13:45:26.0279482Z ---------------------------------------------------------------------- 2022-08-17T13:45:26.2964166Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27232 2022-08-17T13:45:26.2993562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27233 2022-08-17T13:45:26.3022548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27234 2022-08-17T13:45:27.3008243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:27.3008897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:27.3010374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:27.3011004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:27.3067333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:27.3067957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:27.3071280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:27.3071967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:27.3120742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:27.3121342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:27.3125551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:27.3126402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:27.6837821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:27.6930382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:27.6930854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:27.6931501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:27.6932047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:27.6938857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:27.7039184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:27.7039796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:27.7952827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:28.1074034Z skip: Need at least 2 CUDA devices (2.079s) 2022-08-17T13:45:28.1074332Z 2022-08-17T13:45:28.1074805Z ---------------------------------------------------------------------- 2022-08-17T13:45:28.1075236Z Ran 1 test in 2.079s 2022-08-17T13:45:28.1075406Z 2022-08-17T13:45:28.1075519Z OK (skipped=1) 2022-08-17T13:45:28.1075699Z 2022-08-17T13:45:28.1075842Z Generating XML reports... 2022-08-17T13:45:28.1112069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134526.xml 2022-08-17T13:45:29.4008879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:29.4009272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:29.4012252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:29.4012643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:29.7531457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:29.7541140Z 2022-08-17T13:45:29.7541287Z Running tests... 2022-08-17T13:45:29.7541873Z ---------------------------------------------------------------------- 2022-08-17T13:45:30.0161478Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27285 2022-08-17T13:45:30.0190437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27286 2022-08-17T13:45:30.0221445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27287 2022-08-17T13:45:31.0013561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:31.0014247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:31.0018134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:31.0018802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:31.0104060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:31.0104725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:31.0107956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:31.0108658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:31.0628887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:31.0629421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:31.0632274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:31.0632818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:31.4303984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:31.4404580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:31.4405202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:31.4406646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:31.4407515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:31.4408468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:31.4511581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:31.5418721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:31.5419308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:31.8274020Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:45:31.8274340Z 2022-08-17T13:45:31.8274726Z ---------------------------------------------------------------------- 2022-08-17T13:45:31.8275007Z Ran 1 test in 2.073s 2022-08-17T13:45:31.8275127Z 2022-08-17T13:45:31.8275206Z OK (skipped=1) 2022-08-17T13:45:31.8275323Z 2022-08-17T13:45:31.8275413Z Generating XML reports... 2022-08-17T13:45:31.8309526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134529.xml 2022-08-17T13:45:33.1286413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:33.1286817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:33.1289346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:33.1289750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:33.4744325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:33.4754158Z 2022-08-17T13:45:33.4754297Z Running tests... 2022-08-17T13:45:33.4754761Z ---------------------------------------------------------------------- 2022-08-17T13:45:33.7451238Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27338 2022-08-17T13:45:33.7481069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27339 2022-08-17T13:45:33.7513735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27340 2022-08-17T13:45:34.7499088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:34.7499492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:34.7502150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:34.7502744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:34.7517150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:34.7517819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:34.7522278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:34.7522970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:34.7718447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:34.7719149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:34.7722303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:34.7722890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:35.1299188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:35.1401766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:35.1402602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:35.1403206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:35.1404055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:35.1404780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:35.1410232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:35.1411362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:35.1411890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:35.4562940Z skip: Need at least 2 CUDA devices (1.980s) 2022-08-17T13:45:35.4563141Z 2022-08-17T13:45:35.4563705Z ---------------------------------------------------------------------- 2022-08-17T13:45:35.4563996Z Ran 1 test in 1.981s 2022-08-17T13:45:35.4564118Z 2022-08-17T13:45:35.4564195Z OK (skipped=1) 2022-08-17T13:45:35.4564310Z 2022-08-17T13:45:35.4564400Z Generating XML reports... 2022-08-17T13:45:35.4599477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134533.xml 2022-08-17T13:45:36.7513520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:36.7513923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:36.7516284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:36.7516667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:37.1030941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:37.1041255Z 2022-08-17T13:45:37.1041626Z Running tests... 2022-08-17T13:45:37.1042159Z ---------------------------------------------------------------------- 2022-08-17T13:45:37.3718566Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27391 2022-08-17T13:45:37.3748444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27392 2022-08-17T13:45:37.3776775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27393 2022-08-17T13:45:38.3711898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:38.3712530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:38.3715078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:38.3715716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:38.3771526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:38.3772183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:38.3775381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:38.3776072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:38.4030891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:38.4031856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:38.4034335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:38.4034979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:38.7607463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:38.7642769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:38.7643192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:38.7643852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:38.7644397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:38.7707972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:38.7750834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:38.7751868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:38.8721754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:39.1826275Z skip: Need at least 4 CUDA devices (2.078s) 2022-08-17T13:45:39.1826605Z 2022-08-17T13:45:39.1827119Z ---------------------------------------------------------------------- 2022-08-17T13:45:39.1827373Z Ran 1 test in 2.078s 2022-08-17T13:45:39.1827500Z 2022-08-17T13:45:39.1827579Z OK (skipped=1) 2022-08-17T13:45:39.1827893Z 2022-08-17T13:45:39.1828218Z Generating XML reports... 2022-08-17T13:45:39.1865151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134537.xml 2022-08-17T13:45:40.4934112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:40.4934508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:40.4936850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:40.4937295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:40.8358397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:40.8368744Z 2022-08-17T13:45:40.8368844Z Running tests... 2022-08-17T13:45:40.8369428Z ---------------------------------------------------------------------- 2022-08-17T13:45:41.1025830Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27444 2022-08-17T13:45:41.1055141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27445 2022-08-17T13:45:41.1083212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27446 2022-08-17T13:45:42.0963184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:42.0963833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:42.0965118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:42.0965747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:42.0972863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:42.0973480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:42.0977309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:42.0977954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:42.1096945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:42.1097333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:42.1100832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:42.1101474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:42.4752267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:42.4854107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:42.4855166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:42.4855772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:42.4856509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:42.4857440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:42.4862807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:42.4863558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:42.4864460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:42.8131199Z skip: Need at least 4 CUDA devices (1.976s) 2022-08-17T13:45:42.8131423Z 2022-08-17T13:45:42.8131954Z ---------------------------------------------------------------------- 2022-08-17T13:45:42.8132426Z Ran 1 test in 1.976s 2022-08-17T13:45:42.8132549Z 2022-08-17T13:45:42.8132612Z OK (skipped=1) 2022-08-17T13:45:42.8132726Z 2022-08-17T13:45:42.8132823Z Generating XML reports... 2022-08-17T13:45:42.8168197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134540.xml 2022-08-17T13:45:44.1089464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:44.1089862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:44.1092319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:44.1092680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:44.4510143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:44.4520401Z 2022-08-17T13:45:44.4520896Z Running tests... 2022-08-17T13:45:44.4521387Z ---------------------------------------------------------------------- 2022-08-17T13:45:44.7127677Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27497 2022-08-17T13:45:44.7156749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27498 2022-08-17T13:45:44.7186270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27499 2022-08-17T13:45:45.7144157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:45.7144820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:45.7147199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:45.7148185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:45.7282317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:45.7283006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:45.7286096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:45.7286714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:45.7362112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:45.7362512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:45.7365302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:45.7365747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:46.1009366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:46.1109878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:46.1110933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:46.1112186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:46.1113371Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:46.1114518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:46.1218215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:46.1218639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:46.2125305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:46.5234148Z skip: Need at least 2 CUDA devices (2.071s) 2022-08-17T13:45:46.5234466Z 2022-08-17T13:45:46.5234859Z ---------------------------------------------------------------------- 2022-08-17T13:45:46.5235127Z Ran 1 test in 2.071s 2022-08-17T13:45:46.5235247Z 2022-08-17T13:45:46.5235311Z OK (skipped=1) 2022-08-17T13:45:46.5235429Z 2022-08-17T13:45:46.5235518Z Generating XML reports... 2022-08-17T13:45:46.5270207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134544.xml 2022-08-17T13:45:47.8259411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:47.8259840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:47.8262318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:47.8262697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:48.1747495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:48.1757475Z 2022-08-17T13:45:48.1757597Z Running tests... 2022-08-17T13:45:48.1758226Z ---------------------------------------------------------------------- 2022-08-17T13:45:48.4501796Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27550 2022-08-17T13:45:48.4531380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27551 2022-08-17T13:45:48.4560270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27552 2022-08-17T13:45:49.4396167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:49.4397105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:49.4399836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:49.4400253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:49.4738355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:49.4738765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:49.4741744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:49.4742236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:49.4880435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:49.4880816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:49.4884164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:49.4884520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:49.8564694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:49.8665551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:49.8665996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:49.8666632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:49.8667196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:49.8667763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:49.8773178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:49.9677101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:49.9677518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:49.9886254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:45:49.9986988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:45:49.9987796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:45:49.9989100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:49.9990142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:49.9991317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:49.9992285Z STAGE:2022-08-17 13:45:49 27552:27552 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:49.9993516Z STAGE:2022-08-17 13:45:49 27550:27550 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:45:49 27551:27551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:49.9993960Z 2022-08-17T13:45:49.9997604Z STAGE:2022-08-17 13:45:49 27552:27552 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:49.9998474Z STAGE:2022-08-17 13:45:49 27551:27551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:49.9999434Z STAGE:2022-08-17 13:45:49 27550:27550 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.0007405Z STAGE:2022-08-17 13:45:50 27552:27552 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:50.0007928Z STAGE:2022-08-17 13:45:50 27551:27551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:50.0013259Z STAGE:2022-08-17 13:45:50 27550:27550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:50.0018181Z STAGE:2022-08-17 13:45:50 27552:27552 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.0018956Z STAGE:2022-08-17 13:45:50 27550:27550 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.0019799Z STAGE:2022-08-17 13:45:50 27551:27551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.0025061Z STAGE:2022-08-17 13:45:50 27550:27550 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:45:50 27552:27552 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:50.0025389Z 2022-08-17T13:45:50.0030626Z STAGE:2022-08-17 13:45:50 27551:27551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:50.0035932Z STAGE:2022-08-17 13:45:50 27550:27550 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:45:50 27551:27551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.0036414Z 2022-08-17T13:45:50.0036890Z STAGE:2022-08-17 13:45:50 27552:27552 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:50.4613308Z ok (2.285s) 2022-08-17T13:45:50.4613559Z 2022-08-17T13:45:50.4614101Z ---------------------------------------------------------------------- 2022-08-17T13:45:50.4614534Z Ran 1 test in 2.286s 2022-08-17T13:45:50.4614649Z 2022-08-17T13:45:50.4614721Z OK 2022-08-17T13:45:50.4614821Z 2022-08-17T13:45:50.4614917Z Generating XML reports... 2022-08-17T13:45:50.4652348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134548.xml 2022-08-17T13:45:51.7679754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:51.7680171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:51.7682736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:51.7683101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:52.1150837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:52.1160729Z 2022-08-17T13:45:52.1160853Z Running tests... 2022-08-17T13:45:52.1161311Z ---------------------------------------------------------------------- 2022-08-17T13:45:52.3813359Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27615 2022-08-17T13:45:52.3841685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27616 2022-08-17T13:45:52.3870720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27617 2022-08-17T13:45:53.3660868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:53.3661524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:53.3664465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:53.3664871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:53.3906156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:53.3908725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:53.3909750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:53.3910128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:53.4407468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:53.4408053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:53.4412503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:53.4413057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:53.8146834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:53.8248037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:53.8248531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:53.8249193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:53.8249908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:53.8250470Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:53.8356104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:53.9260831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:53.9261239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:53.9569209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:45:53.9570001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:45:53.9570540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:45:53.9571493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:53.9572470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:53.9573299Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:53.9574077Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9574578Z STAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9575036Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9580527Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9581174Z STAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9581668Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9590242Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9590723Z STAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9596026Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9600523Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9601213Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9601675Z STAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9607498Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9608299Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9613981Z STAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:53.9618782Z STAGE:2022-08-17 13:45:53 27617:27617 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9619639Z STAGE:2022-08-17 13:45:53 27615:27615 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:45:53 27616:27616 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:53.9619951Z 2022-08-17T13:45:54.3922031Z ok (2.276s) 2022-08-17T13:45:54.3922203Z 2022-08-17T13:45:54.3922666Z ---------------------------------------------------------------------- 2022-08-17T13:45:54.3923012Z Ran 1 test in 2.276s 2022-08-17T13:45:54.3923138Z 2022-08-17T13:45:54.3923189Z OK 2022-08-17T13:45:54.3923284Z 2022-08-17T13:45:54.3923597Z Generating XML reports... 2022-08-17T13:45:54.3958935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134552.xml 2022-08-17T13:45:55.6990029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:55.6990419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:55.6992819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:55.6993182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:56.0437801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:56.0447958Z 2022-08-17T13:45:56.0448172Z Running tests... 2022-08-17T13:45:56.0448805Z ---------------------------------------------------------------------- 2022-08-17T13:45:56.3083033Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27680 2022-08-17T13:45:56.3110666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27681 2022-08-17T13:45:56.3138810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27682 2022-08-17T13:45:57.3093943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:57.3094620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:57.3097486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:57.3098173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:57.3151664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:57.3152260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:57.3156656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:57.3157277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:57.3254261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:57.3254897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:57.3258275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:57.3259248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:57.6936063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:45:57.7037089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:45:57.7037773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:45:57.7038760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:57.7039406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:57.7039977Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:45:57.7045610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:45:57.7047055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:45:57.7050829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:45:57.7154302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:45:57.7255604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:45:57.7256198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:45:57.7257130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:57.7258101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:57.7258954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:45:57.7259775Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7260516Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7261230Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7266026Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7266651Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7267109Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7274532Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7274998Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7281425Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7285465Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7290339Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7290841Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7292493Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7298330Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7302581Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:45:57.7307537Z STAGE:2022-08-17 13:45:57 27680:27680 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7308235Z STAGE:2022-08-17 13:45:57 27681:27681 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:57.7308703Z STAGE:2022-08-17 13:45:57 27682:27682 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:45:58.1189478Z ok (2.074s) 2022-08-17T13:45:58.1189746Z 2022-08-17T13:45:58.1190263Z ---------------------------------------------------------------------- 2022-08-17T13:45:58.1190565Z Ran 1 test in 2.074s 2022-08-17T13:45:58.1190689Z 2022-08-17T13:45:58.1190756Z OK 2022-08-17T13:45:58.1190851Z 2022-08-17T13:45:58.1190933Z Generating XML reports... 2022-08-17T13:45:58.1225339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134556.xml 2022-08-17T13:45:59.4219368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:45:59.4219839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:45:59.4222399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:45:59.4223132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:45:59.7624984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:45:59.7634862Z 2022-08-17T13:45:59.7634991Z Running tests... 2022-08-17T13:45:59.7636189Z ---------------------------------------------------------------------- 2022-08-17T13:46:00.0232145Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27745 2022-08-17T13:46:00.0261182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27746 2022-08-17T13:46:00.0291076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27747 2022-08-17T13:46:01.0012143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:01.0012816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:01.0015151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:01.0015805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:01.0543488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:01.0544169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:01.0545776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:01.0546399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:01.0557753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:01.0558412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:01.0562471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:01.0563121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:01.4188339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:01.4289232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:01.4289860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:01.4290544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:01.4291358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:01.4292275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:01.4298199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:01.4298840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:01.4299825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:01.4606997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:01.4607788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:01.4608275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:01.4608930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:01.4609492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:01.4610300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:01.4611135Z STAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4711995Z STAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4712327Z 2022-08-17T13:46:01.4717816Z STAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4718361Z 2022-08-17T13:46:01.4718366Z 2022-08-17T13:46:01.4730430Z STAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4730927Z 2022-08-17T13:46:01.4732703Z STAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4737420Z STAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4738001Z STAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4738464Z STAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4746355Z STAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4746683Z 2022-08-17T13:46:01.4749941Z STAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:01.4754165Z STAGE:2022-08-17 13:46:01 27746:27746 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4755061Z STAGE:2022-08-17 13:46:01 27745:27745 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.4755527Z STAGE:2022-08-17 13:46:01 27747:27747 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:01.9343628Z ok (2.171s) 2022-08-17T13:46:01.9343856Z 2022-08-17T13:46:01.9344338Z ---------------------------------------------------------------------- 2022-08-17T13:46:01.9344946Z Ran 1 test in 2.171s 2022-08-17T13:46:01.9345148Z 2022-08-17T13:46:01.9345600Z OK 2022-08-17T13:46:01.9345755Z 2022-08-17T13:46:01.9345859Z Generating XML reports... 2022-08-17T13:46:01.9382815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134559.xml 2022-08-17T13:46:03.2420659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:03.2421067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:03.2423438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:03.2423983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:03.5851880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:03.5861735Z 2022-08-17T13:46:03.5861999Z Running tests... 2022-08-17T13:46:03.5862412Z ---------------------------------------------------------------------- 2022-08-17T13:46:03.8546443Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27810 2022-08-17T13:46:03.8573693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27811 2022-08-17T13:46:03.8602568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27812 2022-08-17T13:46:04.8314294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:04.8314995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:04.8318429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:04.8319101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:04.8897915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:04.8898582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:04.8901202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:04.8901801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:04.8932248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:04.8932794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:04.8936785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:04.8937355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:05.2592313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:05.2692566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:05.2693225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:05.2694209Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:05.2695157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:05.2696075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:05.2801424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:05.2801877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:05.3007090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:05.3007715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:05.3705016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:05.3707190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:05.3708129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:05.3715708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:05.3716444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:05.3718603Z STAGE:2022-08-17 13:46:05 27811:27811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:05.3719740Z STAGE:2022-08-17 13:46:05 27812:27812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:05.3726432Z STAGE:2022-08-17 13:46:05 27811:27811 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:05.3728103Z STAGE:2022-08-17 13:46:05 27812:27812 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:05.3737487Z STAGE:2022-08-17 13:46:05 27812:27812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:05.3741292Z STAGE:2022-08-17 13:46:05 27811:27811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:05.3747311Z STAGE:2022-08-17 13:46:05 27811:27811 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:05 27812:27812 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:05.3747816Z 2022-08-17T13:46:05.8652685Z ok (2.279s) 2022-08-17T13:46:05.8652879Z 2022-08-17T13:46:05.8653330Z ---------------------------------------------------------------------- 2022-08-17T13:46:05.8653844Z Ran 1 test in 2.279s 2022-08-17T13:46:05.8654031Z 2022-08-17T13:46:05.8654096Z OK 2022-08-17T13:46:05.8654196Z 2022-08-17T13:46:05.8654292Z Generating XML reports... 2022-08-17T13:46:05.8714706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134603.xml 2022-08-17T13:46:07.1808507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:07.1808950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:07.1811337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:07.1811714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:07.5239204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:07.5249224Z 2022-08-17T13:46:07.5249393Z Running tests... 2022-08-17T13:46:07.5249840Z ---------------------------------------------------------------------- 2022-08-17T13:46:07.7901756Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27871 2022-08-17T13:46:07.7931159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27872 2022-08-17T13:46:07.7960010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27873 2022-08-17T13:46:08.7802481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:08.7803110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:08.7806305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:08.7806927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:08.8088890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:08.8089537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:08.8092159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:08.8092831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:08.8161866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:08.8162528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:08.8165441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:08.8166296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:09.1921843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:09.2023536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:09.2024477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:09.2025125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:09.2025661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:09.2026200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:09.2131043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:09.3036732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:09.3037359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:09.3038183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:09.3143620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:09.3144727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:09.3145315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:09.3146030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:09.3148066Z STAGE:2022-08-17 13:46:09 27872:27872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:09.3243002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:09.3248179Z STAGE:2022-08-17 13:46:09 27873:27873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:09.3254841Z STAGE:2022-08-17 13:46:09 27873:27873 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:09.3255458Z STAGE:2022-08-17 13:46:09 27872:27872 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:09.3264598Z STAGE:2022-08-17 13:46:09 27873:27873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:09.3270269Z STAGE:2022-08-17 13:46:09 27872:27872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:09.3275110Z STAGE:2022-08-17 13:46:09 27872:27872 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:09.3275622Z STAGE:2022-08-17 13:46:09 27873:27873 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:09.8014080Z ok (2.276s) 2022-08-17T13:46:09.8014651Z 2022-08-17T13:46:09.8014976Z ---------------------------------------------------------------------- 2022-08-17T13:46:09.8015257Z Ran 1 test in 2.276s 2022-08-17T13:46:09.8015379Z 2022-08-17T13:46:09.8015429Z OK 2022-08-17T13:46:09.8015526Z 2022-08-17T13:46:09.8015621Z Generating XML reports... 2022-08-17T13:46:09.8050342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134607.xml 2022-08-17T13:46:11.0994383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:11.0994841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:11.0996645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:11.0997169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:11.4362784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:11.4371737Z 2022-08-17T13:46:11.4371872Z Running tests... 2022-08-17T13:46:11.4372488Z ---------------------------------------------------------------------- 2022-08-17T13:46:11.6987594Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27932 2022-08-17T13:46:11.7017424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27933 2022-08-17T13:46:11.7046452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27934 2022-08-17T13:46:12.7415864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:12.7416510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:12.7419229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:12.7419914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:12.7544576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:12.7545211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:12.7547717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:12.7548366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:12.7754777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:12.7755168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:12.7758355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:12.7758874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:13.1481547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:13.1485539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:13.1486203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:13.1486894Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:13.1487454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:13.1582334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:13.1594179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:13.1595287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:13.1803757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:13.1804177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:13.2594233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:13.2596613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:13.2597488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:13.2614825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:13.2615512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:13.2617997Z STAGE:2022-08-17 13:46:13 27933:27933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:13.2618522Z STAGE:2022-08-17 13:46:13 27934:27934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:13.2626160Z STAGE:2022-08-17 13:46:13 27934:27934 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:13.2626651Z STAGE:2022-08-17 13:46:13 27933:27933 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:13.2635460Z STAGE:2022-08-17 13:46:13 27934:27934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:13.2640483Z STAGE:2022-08-17 13:46:13 27933:27933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:13.2645002Z STAGE:2022-08-17 13:46:13 27933:27933 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:13.2645507Z STAGE:2022-08-17 13:46:13 27934:27934 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:13.7100535Z ok (2.273s) 2022-08-17T13:46:13.7100752Z 2022-08-17T13:46:13.7101218Z ---------------------------------------------------------------------- 2022-08-17T13:46:13.7101644Z Ran 1 test in 2.273s 2022-08-17T13:46:13.7101838Z 2022-08-17T13:46:13.7101922Z OK 2022-08-17T13:46:13.7102071Z 2022-08-17T13:46:13.7102236Z Generating XML reports... 2022-08-17T13:46:13.7137328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134611.xml 2022-08-17T13:46:15.0061791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:15.0062226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:15.0064751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:15.0065131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:15.3523788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:15.3533825Z 2022-08-17T13:46:15.3534115Z Running tests... 2022-08-17T13:46:15.3534743Z ---------------------------------------------------------------------- 2022-08-17T13:46:15.6235123Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27993 2022-08-17T13:46:15.6263342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27994 2022-08-17T13:46:15.6292205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27995 2022-08-17T13:46:16.6189701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:16.6190366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:16.6192077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:16.6193019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:16.6203845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:16.6204431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:16.6208748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:16.6209305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:16.6210098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:16.6210701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:16.6213287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:16.6213965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:16.9990096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:17.0092106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:17.0092760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:17.0093393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:17.0093948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:17.0094503Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:17.0099954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:17.0103218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:17.0103690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:17.0104186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:17.0308816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:17.0309555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:17.0310201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:17.0310762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:17.0311370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:17.0313003Z STAGE:2022-08-17 13:46:17 27994:27994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:17.0413139Z STAGE:2022-08-17 13:46:17 27995:27995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:17.0419038Z STAGE:2022-08-17 13:46:17 27995:27995 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:17.0419657Z STAGE:2022-08-17 13:46:17 27994:27994 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:17.0428050Z STAGE:2022-08-17 13:46:17 27995:27995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:17.0433905Z STAGE:2022-08-17 13:46:17 27994:27994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:17.0438417Z STAGE:2022-08-17 13:46:17 27995:27995 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:17 27994:27994 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:17.0438912Z 2022-08-17T13:46:17.4340311Z ok (2.080s) 2022-08-17T13:46:17.4340576Z 2022-08-17T13:46:17.4341092Z ---------------------------------------------------------------------- 2022-08-17T13:46:17.4341557Z Ran 1 test in 2.081s 2022-08-17T13:46:17.4341691Z 2022-08-17T13:46:17.4341755Z OK 2022-08-17T13:46:17.4341853Z 2022-08-17T13:46:17.4341950Z Generating XML reports... 2022-08-17T13:46:17.4377169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134615.xml 2022-08-17T13:46:18.7507252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:18.7507997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:18.7529327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:18.7529776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:19.0956333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:19.0966223Z 2022-08-17T13:46:19.0966345Z Running tests... 2022-08-17T13:46:19.0967262Z ---------------------------------------------------------------------- 2022-08-17T13:46:19.3691297Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28054 2022-08-17T13:46:19.3722097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28055 2022-08-17T13:46:19.3751379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28056 2022-08-17T13:46:20.3400334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:20.3401018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:20.3403986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:20.3404580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:20.3682664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:20.3683338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:20.3686173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:20.3686825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:20.4183081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:20.4183499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:20.4186712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:20.4187130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:20.7779823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:20.7880839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:20.7881463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:20.7882191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:20.7882961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:20.7883886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:20.7989090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:20.7992841Z STAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8894192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:20.8894768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:20.8898579Z STAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8899065Z STAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8905466Z STAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8906025Z 2022-08-17T13:46:20.8906443Z STAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8914840Z STAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8915864Z STAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8921252Z STAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8925621Z STAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8926678Z STAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8926989Z 2022-08-17T13:46:20.8931320Z STAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8932176Z STAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8938381Z STAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:20.8943285Z STAGE:2022-08-17 13:46:20 28055:28055 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8943925Z STAGE:2022-08-17 13:46:20 28054:28054 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:20 28056:28056 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:20.8944238Z 2022-08-17T13:46:21.2803560Z ok (2.183s) 2022-08-17T13:46:21.2803723Z 2022-08-17T13:46:21.2804188Z ---------------------------------------------------------------------- 2022-08-17T13:46:21.2804668Z Ran 1 test in 2.184s 2022-08-17T13:46:21.2804853Z 2022-08-17T13:46:21.2804909Z OK 2022-08-17T13:46:21.2805008Z 2022-08-17T13:46:21.2805126Z Generating XML reports... 2022-08-17T13:46:21.2842549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134619.xml 2022-08-17T13:46:22.5949637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:22.5950367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:22.5953330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:22.5953912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:22.9480808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:22.9491685Z 2022-08-17T13:46:22.9492147Z Running tests... 2022-08-17T13:46:22.9492586Z ---------------------------------------------------------------------- 2022-08-17T13:46:23.2219510Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28110 2022-08-17T13:46:23.2249836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28111 2022-08-17T13:46:23.2278675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28112 2022-08-17T13:46:24.2154769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:24.2155399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:24.2156833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:24.2157458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:24.2158284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:24.2158894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:24.2162065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:24.2162721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:24.2170596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:24.2171230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:24.2175196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:24.2175875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:24.6081060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:24.6182389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:24.6183026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:24.6183697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:24.6184256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:24.6184806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:24.6291010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:24.6293896Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7196528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:24.7196930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:24.7201158Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7201651Z 2022-08-17T13:46:24.7208213Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7208550Z 2022-08-17T13:46:24.7208832Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7217670Z STAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7218207Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7224322Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7229510Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7230226Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7230704Z STAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7236032Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7236536Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7242128Z STAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:24.7248144Z STAGE:2022-08-17 13:46:24 28111:28111 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7248616Z STAGE:2022-08-17 13:46:24 28110:28110 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:24.7249082Z STAGE:2022-08-17 13:46:24 28112:28112 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:25.1328522Z ok (2.183s) 2022-08-17T13:46:25.1328783Z 2022-08-17T13:46:25.1329532Z ---------------------------------------------------------------------- 2022-08-17T13:46:25.1329963Z Ran 1 test in 2.184s 2022-08-17T13:46:25.1330160Z 2022-08-17T13:46:25.1330241Z OK 2022-08-17T13:46:25.1330400Z 2022-08-17T13:46:25.1330544Z Generating XML reports... 2022-08-17T13:46:25.1366077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134622.xml 2022-08-17T13:46:26.4390270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:26.4390663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:26.4393409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:26.4393918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:26.7877177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:26.7886897Z 2022-08-17T13:46:26.7887332Z Running tests... 2022-08-17T13:46:26.7887988Z ---------------------------------------------------------------------- 2022-08-17T13:46:26.7907977Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-08-17T13:46:26.7908364Z 2022-08-17T13:46:26.7908722Z ---------------------------------------------------------------------- 2022-08-17T13:46:26.7909015Z Ran 1 test in 0.002s 2022-08-17T13:46:26.7909176Z 2022-08-17T13:46:26.7909241Z OK (skipped=1) 2022-08-17T13:46:26.7909360Z 2022-08-17T13:46:26.7909450Z Generating XML reports... 2022-08-17T13:46:26.7935149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134626.xml 2022-08-17T13:46:27.9511998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:27.9512386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:27.9514832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:27.9515221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:28.2966712Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:28.2976718Z 2022-08-17T13:46:28.2976857Z Running tests... 2022-08-17T13:46:28.2977852Z ---------------------------------------------------------------------- 2022-08-17T13:46:28.5629537Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28176 2022-08-17T13:46:28.5657966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28177 2022-08-17T13:46:28.5686760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28178 2022-08-17T13:46:29.5533388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:29.5534050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:29.5537226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:29.5537941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:29.5538753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:29.5539316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:29.5543675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:29.5544285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:29.5558664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:29.5559328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:29.5563588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:29.5564253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:29.9232316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:29.9332240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:29.9332969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:29.9334121Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:29.9334996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:29.9335550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:29.9440828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:29.9441481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:29.9444852Z STAGE:2022-08-17 13:46:29 28178:28178 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:29.9445339Z STAGE:2022-08-17 13:46:29 28177:28177 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:29.9450860Z STAGE:2022-08-17 13:46:29 28178:28178 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:29.9460001Z STAGE:2022-08-17 13:46:29 28178:28178 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0345370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:30.0348613Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0354559Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0355112Z STAGE:2022-08-17 13:46:30 28177:28177 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0364984Z STAGE:2022-08-17 13:46:30 28177:28177 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0369692Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0375032Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0375795Z STAGE:2022-08-17 13:46:30 28178:28178 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0376541Z STAGE:2022-08-17 13:46:30 28177:28177 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0381151Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0382436Z STAGE:2022-08-17 13:46:30 28178:28178 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0388934Z STAGE:2022-08-17 13:46:30 28177:28177 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:30.0394606Z STAGE:2022-08-17 13:46:30 28177:28177 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0395446Z STAGE:2022-08-17 13:46:30 28176:28176 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.0396274Z STAGE:2022-08-17 13:46:30 28178:28178 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:30.4736459Z ok (2.176s) 2022-08-17T13:46:30.4736721Z 2022-08-17T13:46:30.4737142Z ---------------------------------------------------------------------- 2022-08-17T13:46:30.4737708Z Ran 1 test in 2.176s 2022-08-17T13:46:30.4737836Z 2022-08-17T13:46:30.4737899Z OK 2022-08-17T13:46:30.4737981Z 2022-08-17T13:46:30.4738080Z Generating XML reports... 2022-08-17T13:46:30.4772384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134628.xml 2022-08-17T13:46:31.7970273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:31.7970658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:31.7973252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:31.7973641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:32.1442432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:32.1452440Z 2022-08-17T13:46:32.1452549Z Running tests... 2022-08-17T13:46:32.1453181Z ---------------------------------------------------------------------- 2022-08-17T13:46:32.4188275Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28232 2022-08-17T13:46:32.4220114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28233 2022-08-17T13:46:32.4250445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28234 2022-08-17T13:46:33.4401269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:33.4401903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:33.4404085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:33.4404738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:33.4530249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:33.4530916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:33.4534040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:33.4534627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:33.4946592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:33.4946977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:33.4950271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:33.4950767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:33.8626756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:33.8652625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:33.8653086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:33.8653734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:33.8654288Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:33.8727907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:33.8760274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:33.8760966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:33.8764078Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.8765429Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.8770118Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.8780279Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9739206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:33.9743280Z STAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9749254Z STAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9749845Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9758351Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9763828Z STAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9769022Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9769746Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9770057Z 2022-08-17T13:46:33.9776056Z STAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9776838Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9782694Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:33.9788110Z STAGE:2022-08-17 13:46:33 28232:28232 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9788748Z STAGE:2022-08-17 13:46:33 28233:28233 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:33.9789208Z STAGE:2022-08-17 13:46:33 28234:28234 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:34.4305473Z ok (2.285s) 2022-08-17T13:46:34.4305696Z 2022-08-17T13:46:34.4306209Z ---------------------------------------------------------------------- 2022-08-17T13:46:34.4306608Z Ran 1 test in 2.285s 2022-08-17T13:46:34.4306729Z 2022-08-17T13:46:34.4306794Z OK 2022-08-17T13:46:34.4306889Z 2022-08-17T13:46:34.4307177Z Generating XML reports... 2022-08-17T13:46:34.4341919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134632.xml 2022-08-17T13:46:35.7313696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:35.7314110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:35.7317067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:35.7317502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:36.0712011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:36.0721354Z 2022-08-17T13:46:36.0721511Z Running tests... 2022-08-17T13:46:36.0722165Z ---------------------------------------------------------------------- 2022-08-17T13:46:36.0744531Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-08-17T13:46:36.0744812Z 2022-08-17T13:46:36.0745089Z ---------------------------------------------------------------------- 2022-08-17T13:46:36.0745335Z Ran 1 test in 0.002s 2022-08-17T13:46:36.0745458Z 2022-08-17T13:46:36.0745533Z OK (skipped=1) 2022-08-17T13:46:36.0745865Z 2022-08-17T13:46:36.0745956Z Generating XML reports... 2022-08-17T13:46:36.0771728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134636.xml 2022-08-17T13:46:37.2142786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:37.2143181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:37.2145754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:37.2146125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:37.5554475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:37.5564370Z 2022-08-17T13:46:37.5564471Z Running tests... 2022-08-17T13:46:37.5564988Z ---------------------------------------------------------------------- 2022-08-17T13:46:37.5585982Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-08-17T13:46:37.5586440Z 2022-08-17T13:46:37.5586718Z ---------------------------------------------------------------------- 2022-08-17T13:46:37.5587033Z Ran 1 test in 0.002s 2022-08-17T13:46:37.5587152Z 2022-08-17T13:46:37.5587232Z OK (skipped=1) 2022-08-17T13:46:37.5587345Z 2022-08-17T13:46:37.5587435Z Generating XML reports... 2022-08-17T13:46:37.5613123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134637.xml 2022-08-17T13:46:38.7209931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:38.7210354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:38.7213313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:38.7213737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:39.0709533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:39.0718460Z 2022-08-17T13:46:39.0718608Z Running tests... 2022-08-17T13:46:39.0719222Z ---------------------------------------------------------------------- 2022-08-17T13:46:39.3366431Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28308 2022-08-17T13:46:39.3395099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28309 2022-08-17T13:46:39.3423150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28310 2022-08-17T13:46:40.3319377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:40.3320048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:40.3323541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:40.3323958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:40.3431139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:40.3431612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:40.3434801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:40.3435176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:40.3854169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:40.3854892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:40.3857996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:40.3858714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:40.7517701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:40.7618792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:40.7619549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:40.7620193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:40.7620761Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:40.7621314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:40.7726899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:40.7730840Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8632465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:40.8633109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:40.8636639Z STAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8637249Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8645585Z STAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8646326Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8646831Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8657608Z STAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8662960Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8670579Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8677466Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8678260Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8678702Z STAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8685543Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8686372Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8693459Z STAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:40.8702257Z STAGE:2022-08-17 13:46:40 28308:28308 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:40 28309:28309 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:40.8702602Z 2022-08-17T13:46:40.8702981Z STAGE:2022-08-17 13:46:40 28310:28310 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:41.2472550Z ok (2.175s) 2022-08-17T13:46:41.2472792Z 2022-08-17T13:46:41.2473238Z ---------------------------------------------------------------------- 2022-08-17T13:46:41.2473630Z Ran 1 test in 2.175s 2022-08-17T13:46:41.2473812Z 2022-08-17T13:46:41.2473918Z OK 2022-08-17T13:46:41.2480079Z 2022-08-17T13:46:41.2480445Z Generating XML reports... 2022-08-17T13:46:41.2513946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134639.xml 2022-08-17T13:46:42.5513613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:42.5514011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:42.5516322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:42.5516744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:42.8936261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:42.8946592Z 2022-08-17T13:46:42.8946729Z Running tests... 2022-08-17T13:46:42.8947190Z ---------------------------------------------------------------------- 2022-08-17T13:46:43.1634501Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28364 2022-08-17T13:46:43.1663494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28365 2022-08-17T13:46:43.1691987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28366 2022-08-17T13:46:44.1420842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:44.1421249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:44.1424694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:44.1425076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:44.1491545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:44.1492193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:44.1495402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:44.1495843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:44.2022185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:44.2022939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:44.2026051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:44.2026638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:44.5579025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:44.5579672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:44.5580098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:44.5581132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:44.5581807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:44.5582362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:44.5684289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:44.5688418Z STAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6590340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:44.6591149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:44.6594075Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6595897Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6600607Z STAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6601474Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6602005Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6613537Z STAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6614071Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6614714Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6618307Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6618716Z 2022-08-17T13:46:44.6618989Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6624988Z STAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6625794Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6629990Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:44.6630475Z STAGE:2022-08-17 13:46:44 28364:28364 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6630947Z STAGE:2022-08-17 13:46:44 28366:28366 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:44.6631403Z STAGE:2022-08-17 13:46:44 28365:28365 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:45.0742261Z ok (2.179s) 2022-08-17T13:46:45.0742539Z 2022-08-17T13:46:45.0743136Z ---------------------------------------------------------------------- 2022-08-17T13:46:45.0743477Z Ran 1 test in 2.179s 2022-08-17T13:46:45.0743597Z 2022-08-17T13:46:45.0743661Z OK 2022-08-17T13:46:45.0743744Z 2022-08-17T13:46:45.0743841Z Generating XML reports... 2022-08-17T13:46:45.0779026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134642.xml 2022-08-17T13:46:46.3920657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:46.3921346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:46.3924095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:46.3924689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:46.7310826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:46.7321041Z 2022-08-17T13:46:46.7321355Z Running tests... 2022-08-17T13:46:46.7322011Z ---------------------------------------------------------------------- 2022-08-17T13:46:46.9962634Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28420 2022-08-17T13:46:46.9991380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28421 2022-08-17T13:46:47.0019967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28422 2022-08-17T13:46:48.0405435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:48.0406275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:48.0409101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:48.0409775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:48.0484476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:48.0485138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:48.0489166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:48.0489797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:48.1132591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:48.1133003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:48.1136060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:48.1136552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:48.4824240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:48.4824913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:48.4825310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:48.4825938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:48.4826685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:48.4827245Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:48.4931578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:48.5836698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:48.5837359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:48.9071435Z ok (2.175s) 2022-08-17T13:46:48.9071651Z 2022-08-17T13:46:48.9072070Z ---------------------------------------------------------------------- 2022-08-17T13:46:48.9072360Z Ran 1 test in 2.175s 2022-08-17T13:46:48.9072482Z 2022-08-17T13:46:48.9072548Z OK 2022-08-17T13:46:48.9072647Z 2022-08-17T13:46:48.9072744Z Generating XML reports... 2022-08-17T13:46:48.9108561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134646.xml 2022-08-17T13:46:50.2064481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:50.2064878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:50.2067758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:50.2068176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:50.5559984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:50.5570122Z 2022-08-17T13:46:50.5570229Z Running tests... 2022-08-17T13:46:50.5570739Z ---------------------------------------------------------------------- 2022-08-17T13:46:50.8193660Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28473 2022-08-17T13:46:50.8221686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28474 2022-08-17T13:46:50.8251202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28475 2022-08-17T13:46:51.8300446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:51.8301129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:51.8304756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:51.8305170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:51.8384277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:51.8384702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:51.8387795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:51.8388183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:51.8844085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:51.8844775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:51.8848866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:51.8849480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:52.2475463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:52.2576879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:52.2577902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:52.2578933Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:52.2579502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:52.2580049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:52.2684082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:52.2688181Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3589054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:52.3589711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:52.3594103Z STAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3594627Z STAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3601470Z STAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3602308Z STAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3603005Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3615335Z STAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3618422Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3618910Z STAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3625233Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3625894Z 2022-08-17T13:46:52.3625899Z 2022-08-17T13:46:52.3632750Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3633582Z STAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3634414Z STAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:52.3637508Z STAGE:2022-08-17 13:46:52 28473:28473 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3638375Z STAGE:2022-08-17 13:46:52 28474:28474 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.3645249Z STAGE:2022-08-17 13:46:52 28475:28475 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:52.8305554Z ok (2.273s) 2022-08-17T13:46:52.8305755Z 2022-08-17T13:46:52.8306252Z ---------------------------------------------------------------------- 2022-08-17T13:46:52.8306728Z Ran 1 test in 2.273s 2022-08-17T13:46:52.8306937Z 2022-08-17T13:46:52.8307028Z OK 2022-08-17T13:46:52.8307131Z 2022-08-17T13:46:52.8307213Z Generating XML reports... 2022-08-17T13:46:52.8341481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134650.xml 2022-08-17T13:46:54.1333007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:54.1333401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:54.1336165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:54.1336578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:54.4796674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:54.4806597Z 2022-08-17T13:46:54.4807027Z Running tests... 2022-08-17T13:46:54.4807450Z ---------------------------------------------------------------------- 2022-08-17T13:46:54.4826046Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:46:54.4826490Z 2022-08-17T13:46:54.4826768Z ---------------------------------------------------------------------- 2022-08-17T13:46:54.4827015Z Ran 1 test in 0.002s 2022-08-17T13:46:54.4827138Z 2022-08-17T13:46:54.4827215Z OK (skipped=1) 2022-08-17T13:46:54.4827334Z 2022-08-17T13:46:54.4827424Z Generating XML reports... 2022-08-17T13:46:54.4851074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134654.xml 2022-08-17T13:46:55.6304999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:55.6305426Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:55.6307573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:55.6307953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:55.9690068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:55.9700066Z 2022-08-17T13:46:55.9700335Z Running tests... 2022-08-17T13:46:55.9700705Z ---------------------------------------------------------------------- 2022-08-17T13:46:55.9718897Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-08-17T13:46:55.9719383Z 2022-08-17T13:46:55.9719660Z ---------------------------------------------------------------------- 2022-08-17T13:46:55.9719923Z Ran 1 test in 0.002s 2022-08-17T13:46:55.9720044Z 2022-08-17T13:46:55.9720329Z OK (skipped=1) 2022-08-17T13:46:55.9720449Z 2022-08-17T13:46:55.9720540Z Generating XML reports... 2022-08-17T13:46:55.9744615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134655.xml 2022-08-17T13:46:57.1313293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:57.1313689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:57.1315896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:57.1316446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:57.4694239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:46:57.4703638Z 2022-08-17T13:46:57.4703783Z Running tests... 2022-08-17T13:46:57.4704419Z ---------------------------------------------------------------------- 2022-08-17T13:46:57.7369901Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28549 2022-08-17T13:46:57.7399575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28550 2022-08-17T13:46:57.7429184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28551 2022-08-17T13:46:58.7128371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:58.7129014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:58.7131438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:58.7132401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:58.7193551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:58.7193943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:58.7198045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:58.7198424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:58.7526138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:46:58.7526528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:46:58.7529255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:46:58.7530015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:46:59.1083663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:46:59.1184165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:46:59.1184741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:46:59.1185605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:59.1186263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:59.1187173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:46:59.1292076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:46:59.2197732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:46:59.2198131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:46:59.2505684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:46:59.2506464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:46:59.2506937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:46:59.2507601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:59.2508295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:59.2509032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:46:59.2510242Z STAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2510779Z 2022-08-17T13:46:59.2514686Z STAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2525684Z STAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2609238Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2613775Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2614280Z STAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2625078Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2625803Z STAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2629605Z STAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2630663Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2631167Z 2022-08-17T13:46:59.2635758Z STAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2636251Z STAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2636706Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:46:59.2639961Z STAGE:2022-08-17 13:46:59 28549:28549 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2641714Z STAGE:2022-08-17 13:46:59 28550:28550 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.2642512Z STAGE:2022-08-17 13:46:59 28551:28551 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:46:59.6479081Z ok (2.177s) 2022-08-17T13:46:59.6479326Z 2022-08-17T13:46:59.6479847Z ---------------------------------------------------------------------- 2022-08-17T13:46:59.6480190Z Ran 1 test in 2.177s 2022-08-17T13:46:59.6480310Z 2022-08-17T13:46:59.6480373Z OK 2022-08-17T13:46:59.6480469Z 2022-08-17T13:46:59.6480564Z Generating XML reports... 2022-08-17T13:46:59.6518953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134657.xml 2022-08-17T13:47:00.9416322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:00.9416732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:00.9419326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:00.9419941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:01.2837198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:01.2847208Z 2022-08-17T13:47:01.2847296Z Running tests... 2022-08-17T13:47:01.2847792Z ---------------------------------------------------------------------- 2022-08-17T13:47:01.5475655Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28614 2022-08-17T13:47:01.5504175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28615 2022-08-17T13:47:01.5532057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28616 2022-08-17T13:47:02.5404577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:02.5405273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:02.5406913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:02.5407538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:02.5921995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:02.5922700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:02.5925152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:02.5925572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:02.5983228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:02.5983635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:02.5987280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:02.5987767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:02.9739969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:02.9841475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:02.9842737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:02.9843716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:02.9844325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:02.9844879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:02.9851412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:02.9851989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:02.9853616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:02.9855616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:47:02.9959892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:47:02.9960363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:47:02.9961002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:47:02.9961711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:47:02.9963914Z STAGE:2022-08-17 13:47:02 28615:28615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:03.0058989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes. 2022-08-17T13:47:03.0064097Z STAGE:2022-08-17 13:47:03 28616:28616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:03.0069985Z STAGE:2022-08-17 13:47:03 28616:28616 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:03.0070641Z STAGE:2022-08-17 13:47:03 28615:28615 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:03.0081854Z STAGE:2022-08-17 13:47:03 28615:28615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:03.0082336Z STAGE:2022-08-17 13:47:03 28616:28616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:03.0086466Z STAGE:2022-08-17 13:47:03 28616:28616 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:03.0087033Z STAGE:2022-08-17 13:47:03 28615:28615 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:03.4583267Z ok (2.173s) 2022-08-17T13:47:03.4583525Z 2022-08-17T13:47:03.4584045Z ---------------------------------------------------------------------- 2022-08-17T13:47:03.4584301Z Ran 1 test in 2.173s 2022-08-17T13:47:03.4584433Z 2022-08-17T13:47:03.4584499Z OK 2022-08-17T13:47:03.4584595Z 2022-08-17T13:47:03.4584690Z Generating XML reports... 2022-08-17T13:47:03.4619706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134701.xml 2022-08-17T13:47:04.7875985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:04.7876392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:04.7878966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:04.7879346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:05.1254518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:05.1264275Z 2022-08-17T13:47:05.1264425Z Running tests... 2022-08-17T13:47:05.1264895Z ---------------------------------------------------------------------- 2022-08-17T13:47:05.3888421Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28675 2022-08-17T13:47:05.3915795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28676 2022-08-17T13:47:05.3944013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28677 2022-08-17T13:47:06.3848869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:06.3849532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:06.3852392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:06.3853050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:06.4020297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:06.4020989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:06.4024907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:06.4025575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:06.4040353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:06.4041269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:06.4044500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:06.4045154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:06.7707454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:06.7808478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:06.7809259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:06.7809774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:06.7810644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:06.7811489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:06.7816580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:06.7817338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:06.7819168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:07.0992271Z ok (1.972s) 2022-08-17T13:47:07.0992531Z 2022-08-17T13:47:07.0993092Z ---------------------------------------------------------------------- 2022-08-17T13:47:07.0993451Z Ran 1 test in 1.973s 2022-08-17T13:47:07.0993775Z 2022-08-17T13:47:07.0993842Z OK 2022-08-17T13:47:07.0993941Z 2022-08-17T13:47:07.0994052Z Generating XML reports... 2022-08-17T13:47:07.1029105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134705.xml 2022-08-17T13:47:08.3637797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:08.3638457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:08.3641051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:08.3641633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:08.7012432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:08.7022379Z 2022-08-17T13:47:08.7022699Z Running tests... 2022-08-17T13:47:08.7023354Z ---------------------------------------------------------------------- 2022-08-17T13:47:08.9714068Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28728 2022-08-17T13:47:08.9741764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28729 2022-08-17T13:47:08.9769498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28730 2022-08-17T13:47:09.9723129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:09.9723829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:09.9724640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:09.9725186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:09.9726262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:09.9726920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:09.9729708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:09.9730387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:09.9741547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:09.9741971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:09.9746095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:09.9746535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:10.3395620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:10.3495669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:10.3496327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:10.3497256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:10.3498193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:10.3499164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:10.3604877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:10.3605281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:10.4509218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:10.7818876Z ok (2.079s) 2022-08-17T13:47:10.7819119Z 2022-08-17T13:47:10.7819688Z ---------------------------------------------------------------------- 2022-08-17T13:47:10.7819960Z Ran 1 test in 2.080s 2022-08-17T13:47:10.7820086Z 2022-08-17T13:47:10.7820239Z OK 2022-08-17T13:47:10.7820358Z 2022-08-17T13:47:10.7820443Z Generating XML reports... 2022-08-17T13:47:10.7855596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134708.xml 2022-08-17T13:47:12.0969008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:12.0969388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:12.0972178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:12.0972590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:12.4402847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:12.4412489Z 2022-08-17T13:47:12.4412592Z Running tests... 2022-08-17T13:47:12.4413204Z ---------------------------------------------------------------------- 2022-08-17T13:47:12.7078445Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28781 2022-08-17T13:47:12.7107105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28782 2022-08-17T13:47:12.7137208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28783 2022-08-17T13:47:13.6964747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:13.6965175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:13.6968069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:13.6968745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:13.7080660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:13.7081575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:13.7084695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:13.7085324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:13.7544904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:13.7545568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:13.7548683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:13.7549303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:14.1168301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:14.1269198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:14.1269789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:14.1270442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:14.1271003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:14.1271556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:14.1375436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:14.2281481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:14.2282047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:14.5189306Z ok (2.077s) 2022-08-17T13:47:14.5189465Z 2022-08-17T13:47:14.5189991Z ---------------------------------------------------------------------- 2022-08-17T13:47:14.5190391Z Ran 1 test in 2.078s 2022-08-17T13:47:14.5190511Z 2022-08-17T13:47:14.5190575Z OK 2022-08-17T13:47:14.5190672Z 2022-08-17T13:47:14.5190753Z Generating XML reports... 2022-08-17T13:47:14.5226896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134712.xml 2022-08-17T13:47:15.8183123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:15.8183542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:15.8186277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:15.8186658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:16.1610899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:16.1621291Z 2022-08-17T13:47:16.1621707Z Running tests... 2022-08-17T13:47:16.1622147Z ---------------------------------------------------------------------- 2022-08-17T13:47:16.4255801Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28834 2022-08-17T13:47:16.4283447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28835 2022-08-17T13:47:16.4311069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28836 2022-08-17T13:47:17.4152366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:17.4152991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:17.4156287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:17.4156974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:17.4162052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:17.4162690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:17.4166310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:17.4166913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:17.4551825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:17.4552559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:17.4555349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:17.4556082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:17.8151512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:17.8152202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:17.8152832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:17.8153746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:17.8154700Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:17.8155467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:17.8258318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:17.8262468Z STAGE:2022-08-17 13:47:17 28835:28835 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:17.9163925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:17.9164400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:17.9167973Z STAGE:2022-08-17 13:47:17 28834:28834 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:17.9168742Z STAGE:2022-08-17 13:47:17 28836:28836 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:17.9191854Z STAGE:2022-08-17 13:47:17 28834:28834 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:17.9192455Z STAGE:2022-08-17 13:47:17 28835:28835 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:17.9192970Z STAGE:2022-08-17 13:47:17 28836:28836 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:18.3359930Z ok (2.174s) 2022-08-17T13:47:18.3361120Z 2022-08-17T13:47:18.3361446Z ---------------------------------------------------------------------- 2022-08-17T13:47:18.3361710Z Ran 1 test in 2.174s 2022-08-17T13:47:18.3361830Z 2022-08-17T13:47:18.3361880Z OK 2022-08-17T13:47:18.3361977Z 2022-08-17T13:47:18.3362072Z Generating XML reports... 2022-08-17T13:47:18.3396798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134716.xml 2022-08-17T13:47:19.6404561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:19.6404982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:19.6407549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:19.6408172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:19.9883583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:19.9894093Z 2022-08-17T13:47:19.9894196Z Running tests... 2022-08-17T13:47:19.9895379Z ---------------------------------------------------------------------- 2022-08-17T13:47:20.2576351Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28890 2022-08-17T13:47:20.2605793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28891 2022-08-17T13:47:20.2644420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28892 2022-08-17T13:47:21.2462745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:21.2463209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:21.2464593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:21.2465016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:21.2825346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:21.2825735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:21.2829782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:21.2830303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:21.3073191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:21.3073729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:21.3077256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:21.3077711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:21.6665663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:21.6765259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:21.6765738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:21.6766582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:21.6767152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:21.6767691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:21.6873523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:21.6878120Z STAGE:2022-08-17 13:47:21 28892:28892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:21.7778704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:21.7779095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:21.7784680Z STAGE:2022-08-17 13:47:21 28890:28890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:21.7785432Z STAGE:2022-08-17 13:47:21 28891:28891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:21.7808491Z STAGE:2022-08-17 13:47:21 28890:28890 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:21.7809393Z STAGE:2022-08-17 13:47:21 28891:28891 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:21.7810124Z STAGE:2022-08-17 13:47:21 28892:28892 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:22.1697174Z ok (2.180s) 2022-08-17T13:47:22.1697447Z 2022-08-17T13:47:22.1697892Z ---------------------------------------------------------------------- 2022-08-17T13:47:22.1698177Z Ran 1 test in 2.180s 2022-08-17T13:47:22.1698298Z 2022-08-17T13:47:22.1698350Z OK 2022-08-17T13:47:22.1698444Z 2022-08-17T13:47:22.1698543Z Generating XML reports... 2022-08-17T13:47:22.1736694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134719.xml 2022-08-17T13:47:23.4713306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:23.4713720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:23.4716416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:23.4716912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:23.8139031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:23.8148376Z 2022-08-17T13:47:23.8148541Z Running tests... 2022-08-17T13:47:23.8148894Z ---------------------------------------------------------------------- 2022-08-17T13:47:24.0818656Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28946 2022-08-17T13:47:24.0848383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28947 2022-08-17T13:47:24.0876798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28948 2022-08-17T13:47:25.0685791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:25.0686630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:25.0689189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:25.0689779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:25.0759388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:25.0759770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:25.0763928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:25.0764599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:25.1280663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:25.1281208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:25.1284464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:25.1284931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:25.5013046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:25.5013492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:25.5013963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:25.5014669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:25.5015251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:25.5015874Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:25.5022393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:25.5026528Z STAGE:2022-08-17 13:47:25 28947:28947 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:25.6023999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:25.6024382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:25.6027405Z STAGE:2022-08-17 13:47:25 28946:28946 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:25.6028452Z STAGE:2022-08-17 13:47:25 28948:28948 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:25.6053376Z STAGE:2022-08-17 13:47:25 28948:28948 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:25.6054723Z STAGE:2022-08-17 13:47:25 28946:28946 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:25.6055937Z STAGE:2022-08-17 13:47:25 28947:28947 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:25.8927225Z ok (2.078s) 2022-08-17T13:47:25.8927460Z 2022-08-17T13:47:25.8927935Z ---------------------------------------------------------------------- 2022-08-17T13:47:25.8928334Z Ran 1 test in 2.078s 2022-08-17T13:47:25.8928522Z 2022-08-17T13:47:25.8928618Z OK 2022-08-17T13:47:25.8928788Z 2022-08-17T13:47:25.8928944Z Generating XML reports... 2022-08-17T13:47:25.8964072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134723.xml 2022-08-17T13:47:27.1691107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:27.1691768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:27.1693979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:27.1694379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:27.5061762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:27.5071844Z 2022-08-17T13:47:27.5071947Z Running tests... 2022-08-17T13:47:27.5072422Z ---------------------------------------------------------------------- 2022-08-17T13:47:27.5089497Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:47:27.5089889Z 2022-08-17T13:47:27.5090303Z ---------------------------------------------------------------------- 2022-08-17T13:47:27.5090774Z Ran 1 test in 0.002s 2022-08-17T13:47:27.5090941Z 2022-08-17T13:47:27.5091008Z OK (skipped=1) 2022-08-17T13:47:27.5091121Z 2022-08-17T13:47:27.5091208Z Generating XML reports... 2022-08-17T13:47:27.5117479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134727.xml 2022-08-17T13:47:28.6471699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:28.6472126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:28.6475993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:28.6476670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:28.9869499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:28.9879994Z 2022-08-17T13:47:28.9880515Z Running tests... 2022-08-17T13:47:28.9880939Z ---------------------------------------------------------------------- 2022-08-17T13:47:28.9898481Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:47:28.9898728Z 2022-08-17T13:47:28.9900101Z ---------------------------------------------------------------------- 2022-08-17T13:47:28.9900598Z Ran 1 test in 0.002s 2022-08-17T13:47:28.9900725Z 2022-08-17T13:47:28.9900809Z OK (skipped=1) 2022-08-17T13:47:28.9900911Z 2022-08-17T13:47:28.9901004Z Generating XML reports... 2022-08-17T13:47:28.9927284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134728.xml 2022-08-17T13:47:30.1519801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:30.1520181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:30.1522633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:30.1523219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:30.4937999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:30.4947961Z 2022-08-17T13:47:30.4948114Z Running tests... 2022-08-17T13:47:30.4948743Z ---------------------------------------------------------------------- 2022-08-17T13:47:30.4968162Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-08-17T13:47:30.4968589Z 2022-08-17T13:47:30.4969411Z ---------------------------------------------------------------------- 2022-08-17T13:47:30.4969671Z Ran 1 test in 0.002s 2022-08-17T13:47:30.4969798Z 2022-08-17T13:47:30.4969875Z OK (skipped=1) 2022-08-17T13:47:30.4969989Z 2022-08-17T13:47:30.4970080Z Generating XML reports... 2022-08-17T13:47:30.4994445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134730.xml 2022-08-17T13:47:31.6668098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:31.6668497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:31.6671074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:31.6671625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:32.0118991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:32.0129908Z 2022-08-17T13:47:32.0130238Z Running tests... 2022-08-17T13:47:32.0130883Z ---------------------------------------------------------------------- 2022-08-17T13:47:32.2773162Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29032 2022-08-17T13:47:32.2801776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29033 2022-08-17T13:47:32.2829566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29034 2022-08-17T13:47:33.2474235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:33.2474632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:33.2477196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:33.2477902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:33.2775962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:33.2776439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:33.2779821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:33.2780214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:33.2888929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:33.2889289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:33.2893098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:33.2893471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:33.6593247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:33.6593996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:33.6594396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:33.6595117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:33.6595732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:33.6596345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:33.6601162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:33.6601766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:33.6607631Z STAGE:2022-08-17 13:47:33 29033:29033 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:47:33 29034:29034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:33.6607960Z 2022-08-17T13:47:33.7604906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:33.7609918Z STAGE:2022-08-17 13:47:33 29032:29032 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:33.7633869Z STAGE:2022-08-17 13:47:33 29034:29034 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:33.7635445Z STAGE:2022-08-17 13:47:33 29032:29032 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:33.7636732Z STAGE:2022-08-17 13:47:33 29033:29033 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:34.0877620Z ok (2.074s) 2022-08-17T13:47:34.0877903Z 2022-08-17T13:47:34.0878388Z ---------------------------------------------------------------------- 2022-08-17T13:47:34.0878653Z Ran 1 test in 2.075s 2022-08-17T13:47:34.0878774Z 2022-08-17T13:47:34.0878837Z OK 2022-08-17T13:47:34.0878934Z 2022-08-17T13:47:34.0879028Z Generating XML reports... 2022-08-17T13:47:34.0916992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134732.xml 2022-08-17T13:47:35.3766400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:35.3766824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:35.3769610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:35.3769988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:35.7198380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:35.7208340Z 2022-08-17T13:47:35.7208570Z Running tests... 2022-08-17T13:47:35.7208966Z ---------------------------------------------------------------------- 2022-08-17T13:47:35.9893133Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29088 2022-08-17T13:47:35.9920671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29089 2022-08-17T13:47:35.9949760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29090 2022-08-17T13:47:36.9943055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:36.9943460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:36.9945750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:36.9946132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:36.9965144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:36.9965490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:36.9970318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:36.9970798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:37.0278339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:37.0278818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:37.0281619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:37.0282124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:37.3847132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:37.3947927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:37.3948670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:37.3949511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:37.3950077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:37.3950616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:37.4056093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:37.4960911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:37.4961469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:37.8002099Z ok (2.079s) 2022-08-17T13:47:37.8002276Z 2022-08-17T13:47:37.8002671Z ---------------------------------------------------------------------- 2022-08-17T13:47:37.8002953Z Ran 1 test in 2.079s 2022-08-17T13:47:37.8003105Z 2022-08-17T13:47:37.8003205Z OK 2022-08-17T13:47:37.8003302Z 2022-08-17T13:47:37.8003401Z Generating XML reports... 2022-08-17T13:47:37.8047811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134735.xml 2022-08-17T13:47:39.0911781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:39.0912475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:39.0914951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:39.0915562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:39.4464750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:39.4475124Z 2022-08-17T13:47:39.4475474Z Running tests... 2022-08-17T13:47:39.4476131Z ---------------------------------------------------------------------- 2022-08-17T13:47:39.7229691Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29141 2022-08-17T13:47:39.7258327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29142 2022-08-17T13:47:39.7288694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29143 2022-08-17T13:47:40.7469603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:40.7470256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:40.7473193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:40.7473831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:40.7478479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:40.7479097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:40.7482994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:40.7483640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:40.7657917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:40.7658598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:40.7662008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:40.7662688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:41.1514046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:41.1614314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:41.1615213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:41.1616556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:41.1617808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:41.1618519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:41.1623114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:41.1623768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:41.1624460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:41.1627508Z STAGE:2022-08-17 13:47:41 29141:29141 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-08-17 13:47:41 29142:29142 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:41.1627928Z 2022-08-17T13:47:41.1628324Z STAGE:2022-08-17 13:47:41 29143:29143 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:41.1640983Z STAGE:2022-08-17 13:47:41 29141:29141 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:41.1643388Z STAGE:2022-08-17 13:47:41 29142:29142 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:41.1643937Z STAGE:2022-08-17 13:47:41 29143:29143 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:41.5338916Z ok (2.086s) 2022-08-17T13:47:41.5339196Z 2022-08-17T13:47:41.5339656Z ---------------------------------------------------------------------- 2022-08-17T13:47:41.5339917Z Ran 1 test in 2.086s 2022-08-17T13:47:41.5340038Z 2022-08-17T13:47:41.5340104Z OK 2022-08-17T13:47:41.5340199Z 2022-08-17T13:47:41.5340293Z Generating XML reports... 2022-08-17T13:47:41.5375366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134739.xml 2022-08-17T13:47:42.8416591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:42.8416987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:42.8419491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:42.8419856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:43.1889811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:43.1899268Z 2022-08-17T13:47:43.1899349Z Running tests... 2022-08-17T13:47:43.1899907Z ---------------------------------------------------------------------- 2022-08-17T13:47:43.4501983Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29197 2022-08-17T13:47:43.4530036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29198 2022-08-17T13:47:43.4558018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29199 2022-08-17T13:47:44.4392656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:44.4393352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:44.4396698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:44.4397096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:44.4568449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:44.4568885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:44.4571998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:44.4572474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:44.4889096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:44.4889737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:44.4892334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:44.4892986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:44.8438808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:44.8522475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:44.8523259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:44.8523912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:44.8524458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:44.8540207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:44.8630222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:44.8630623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:44.8635922Z STAGE:2022-08-17 13:47:44 29198:29198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:44.8636580Z STAGE:2022-08-17 13:47:44 29199:29199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:44.9551116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:44.9556437Z STAGE:2022-08-17 13:47:44 29197:29197 ActivityProfilerController.cpp:294] Completed Stage: Warm Up 2022-08-17T13:47:44.9573546Z STAGE:2022-08-17 13:47:44 29197:29197 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:44.9576100Z STAGE:2022-08-17 13:47:44 29198:29198 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:44.9576822Z STAGE:2022-08-17 13:47:44 29199:29199 ActivityProfilerController.cpp:300] Completed Stage: Collection 2022-08-17T13:47:45.2606159Z ok (2.070s) 2022-08-17T13:47:45.2606450Z 2022-08-17T13:47:45.2606974Z ---------------------------------------------------------------------- 2022-08-17T13:47:45.2607331Z Ran 1 test in 2.071s 2022-08-17T13:47:45.2607453Z 2022-08-17T13:47:45.2607519Z OK 2022-08-17T13:47:45.2607616Z 2022-08-17T13:47:45.2607712Z Generating XML reports... 2022-08-17T13:47:45.2642095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134743.xml 2022-08-17T13:47:46.5456042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:46.5456415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:46.5458671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:46.5459048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:46.8875598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:46.8886659Z 2022-08-17T13:47:46.8886837Z Running tests... 2022-08-17T13:47:46.8887244Z ---------------------------------------------------------------------- 2022-08-17T13:47:47.1570444Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29253 2022-08-17T13:47:47.1598433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29254 2022-08-17T13:47:47.1626671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29255 2022-08-17T13:47:48.1563227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:48.1563929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:48.1565532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:48.1566387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:48.1795082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:48.1795784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:48.1799235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:48.1799751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:48.1929423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:48.1929801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:48.1933498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:48.1933913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:48.5576827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:48.5677306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:48.5678446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:48.5679784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:48.5680926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:48.5681679Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:48.5784325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:48.6689812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:48.6690413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:49.0675853Z ok (2.179s) 2022-08-17T13:47:49.0676109Z 2022-08-17T13:47:49.0676665Z ---------------------------------------------------------------------- 2022-08-17T13:47:49.0677004Z Ran 1 test in 2.179s 2022-08-17T13:47:49.0677179Z 2022-08-17T13:47:49.0677253Z OK 2022-08-17T13:47:49.0677414Z 2022-08-17T13:47:49.0677521Z Generating XML reports... 2022-08-17T13:47:49.0715605Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134746.xml 2022-08-17T13:47:50.3830833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:50.3831224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:50.3832974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:50.3833584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:50.7234851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:50.7244970Z 2022-08-17T13:47:50.7245110Z Running tests... 2022-08-17T13:47:50.7245707Z ---------------------------------------------------------------------- 2022-08-17T13:47:50.9880770Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29333 2022-08-17T13:47:50.9910591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29334 2022-08-17T13:47:50.9939153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29335 2022-08-17T13:47:51.9797855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:51.9798554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:51.9801305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:51.9801960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:51.9821526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:51.9822135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:51.9823043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:51.9823659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:51.9826358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:51.9826967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:51.9827799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:51.9828400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:52.3600978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:52.3703066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:52.3703697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:52.3704353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:52.3704912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:52.3705469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:52.3711424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:52.3711870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:52.4716641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:52.7988082Z skip: CUDA is not available. (2.074s) 2022-08-17T13:47:52.7988416Z 2022-08-17T13:47:52.7988875Z ---------------------------------------------------------------------- 2022-08-17T13:47:52.7989141Z Ran 1 test in 2.074s 2022-08-17T13:47:52.7989262Z 2022-08-17T13:47:52.7989340Z OK (skipped=1) 2022-08-17T13:47:52.7989454Z 2022-08-17T13:47:52.7989543Z Generating XML reports... 2022-08-17T13:47:52.8024369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134750.xml 2022-08-17T13:47:54.1034449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:54.1034878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:54.1036866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:54.1037245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:54.4538798Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:54.4548640Z 2022-08-17T13:47:54.4548754Z Running tests... 2022-08-17T13:47:54.4549495Z ---------------------------------------------------------------------- 2022-08-17T13:47:54.7205529Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29386 2022-08-17T13:47:54.7236134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29387 2022-08-17T13:47:54.7264850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29388 2022-08-17T13:47:55.7085402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:55.7086331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:55.7088071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:55.7088735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:55.7193899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:55.7194578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:55.7197527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:55.7198210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:55.7259084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:55.7259976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:55.7263036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:55.7263706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:56.0840216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:56.0941166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:56.0941818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:56.0942935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:56.0943807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:56.0944742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:56.1048345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:56.1955141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:56.1955753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:56.5312934Z skip: Need at least 2 CUDA devices (2.076s) 2022-08-17T13:47:56.5313119Z 2022-08-17T13:47:56.5313448Z ---------------------------------------------------------------------- 2022-08-17T13:47:56.5313975Z Ran 1 test in 2.076s 2022-08-17T13:47:56.5314097Z 2022-08-17T13:47:56.5314174Z OK (skipped=1) 2022-08-17T13:47:56.5314288Z 2022-08-17T13:47:56.5314379Z Generating XML reports... 2022-08-17T13:47:56.5349648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134754.xml 2022-08-17T13:47:57.8220640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:57.8221035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:57.8223331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:57.8223864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:58.1579233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:47:58.1589036Z 2022-08-17T13:47:58.1589187Z Running tests... 2022-08-17T13:47:58.1589870Z ---------------------------------------------------------------------- 2022-08-17T13:47:58.4175070Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29439 2022-08-17T13:47:58.4201552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29440 2022-08-17T13:47:58.4229002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29441 2022-08-17T13:47:59.4006647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:59.4007320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:59.4008161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:59.4008802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:59.4009623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:59.4010238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:59.4012257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:47:59.4012899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:47:59.4013697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:59.4014297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:59.4018492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:47:59.4018904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:47:59.7816299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:47:59.7918036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:47:59.7918588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:47:59.7919299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:59.7919952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:59.7920582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:47:59.7927737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:47:59.7928564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:47:59.7993846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeiumjy3 2022-08-17T13:47:59.7994547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm_a3jec 2022-08-17T13:47:59.7995610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeiumjy3/_remote_module_non_scriptable.py 2022-08-17T13:47:59.7996814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm_a3jec/_remote_module_non_scriptable.py 2022-08-17T13:47:59.8929550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:47:59.8994566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu95g6yuj 2022-08-17T13:47:59.8995945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu95g6yuj/_remote_module_non_scriptable.py 2022-08-17T13:48:00.2278118Z ok (2.069s) 2022-08-17T13:48:00.2278366Z 2022-08-17T13:48:00.2278912Z ---------------------------------------------------------------------- 2022-08-17T13:48:00.2279287Z Ran 1 test in 2.069s 2022-08-17T13:48:00.2279407Z 2022-08-17T13:48:00.2279457Z OK 2022-08-17T13:48:00.2279551Z 2022-08-17T13:48:00.2279647Z Generating XML reports... 2022-08-17T13:48:00.2314528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134758.xml 2022-08-17T13:48:01.5070064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:01.5070472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:01.5072610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:01.5073304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:01.8593418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:48:01.8603156Z 2022-08-17T13:48:01.8603359Z Running tests... 2022-08-17T13:48:01.8604011Z ---------------------------------------------------------------------- 2022-08-17T13:48:02.1290260Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29492 2022-08-17T13:48:02.1318322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29493 2022-08-17T13:48:02.1346517Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29494 2022-08-17T13:48:03.1292009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:03.1292668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:03.1294053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:03.1294626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:03.1407139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:03.1407829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:03.1411663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:03.1412288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:03.1454990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:03.1455609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:03.1459352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:03.1460045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:03.5222137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:03.5323503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:03.5324554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:03.5324995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:03.5325566Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:03.5326467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:03.5332679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:03.5333238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:03.5333730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:03.8395780Z skip: Need at least 2 CUDA devices (1.979s) 2022-08-17T13:48:03.8395968Z 2022-08-17T13:48:03.8396391Z ---------------------------------------------------------------------- 2022-08-17T13:48:03.8396672Z Ran 1 test in 1.979s 2022-08-17T13:48:03.8396794Z 2022-08-17T13:48:03.8396872Z OK (skipped=1) 2022-08-17T13:48:03.8397013Z 2022-08-17T13:48:03.8397147Z Generating XML reports... 2022-08-17T13:48:03.8431393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134801.xml 2022-08-17T13:48:05.1155294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:05.1155672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:05.1158816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:05.1159248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:05.4527987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:48:05.4537950Z 2022-08-17T13:48:05.4538077Z Running tests... 2022-08-17T13:48:05.4538950Z ---------------------------------------------------------------------- 2022-08-17T13:48:05.7163472Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29545 2022-08-17T13:48:05.7191984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29546 2022-08-17T13:48:05.7219486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29547 2022-08-17T13:48:06.6921219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:06.6921817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:06.6924682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:06.6925087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:06.7062203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:06.7062987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:06.7066709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:06.7067302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:06.7578928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:06.7579470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:06.7582214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:06.7582761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:07.1210356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:07.1211060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:07.1211605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:07.1212248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:07.1212814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:07.1213348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. 2022-08-17T13:48:07.1317424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:07.1318025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:07.2224031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:07.5270396Z skip: Need at least 2 CUDA devices (2.073s) 2022-08-17T13:48:07.5270888Z 2022-08-17T13:48:07.5271501Z ---------------------------------------------------------------------- 2022-08-17T13:48:07.5271959Z Ran 1 test in 2.073s 2022-08-17T13:48:07.5272175Z 2022-08-17T13:48:07.5272295Z OK (skipped=1) 2022-08-17T13:48:07.5272453Z 2022-08-17T13:48:07.5272530Z Generating XML reports... 2022-08-17T13:48:07.5307270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134805.xml 2022-08-17T13:48:08.7980830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:08.7981545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:08.7985427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:09.1369639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:09.1370571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:48:09.1380589Z 2022-08-17T13:48:09.1380881Z Running tests... 2022-08-17T13:48:09.1381548Z ---------------------------------------------------------------------- 2022-08-17T13:48:09.1399334Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s) 2022-08-17T13:48:09.1399811Z 2022-08-17T13:48:09.1400169Z ---------------------------------------------------------------------- 2022-08-17T13:48:09.1400582Z Ran 1 test in 0.002s 2022-08-17T13:48:09.1400766Z 2022-08-17T13:48:09.1400894Z OK (skipped=1) 2022-08-17T13:48:09.1401078Z 2022-08-17T13:48:09.1401236Z Generating XML reports... 2022-08-17T13:48:09.1426870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134809.xml 2022-08-17T13:48:10.2999223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:10.2999866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:10.3002403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:10.3002982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:10.6489583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-08-17T13:48:10.6500058Z 2022-08-17T13:48:10.6500474Z Running tests... 2022-08-17T13:48:10.6501070Z ---------------------------------------------------------------------- 2022-08-17T13:48:10.6518300Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s) 2022-08-17T13:48:10.6518777Z 2022-08-17T13:48:10.6519147Z ---------------------------------------------------------------------- 2022-08-17T13:48:10.6519562Z Ran 1 test in 0.002s 2022-08-17T13:48:10.6519771Z 2022-08-17T13:48:10.6519907Z OK (skipped=1) 2022-08-17T13:48:10.6520107Z 2022-08-17T13:48:10.6520267Z Generating XML reports... 2022-08-17T13:48:10.6545637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134810.xml 2022-08-17T13:48:11.1193401Z Running distributed/test_launcher ... [2022-08-17 13:48:11.118955] 2022-08-17T13:48:11.1193968Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:48:11.119029] 2022-08-17T13:48:12.5459566Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2022-08-17T13:48:12.5470922Z 2022-08-17T13:48:12.5471235Z Running tests... 2022-08-17T13:48:12.5471759Z ---------------------------------------------------------------------- 2022-08-17T13:48:12.7989498Z test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s) 2022-08-17T13:48:12.7990022Z 2022-08-17T13:48:12.7990238Z ---------------------------------------------------------------------- 2022-08-17T13:48:12.7990480Z Ran 1 test in 0.252s 2022-08-17T13:48:12.7990598Z 2022-08-17T13:48:12.7990682Z OK (skipped=1) 2022-08-17T13:48:12.7990798Z 2022-08-17T13:48:12.7990885Z Generating XML reports... 2022-08-17T13:48:12.8019179Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220817134812.xml 2022-08-17T13:48:13.0877631Z Running distributed/test_nccl ... [2022-08-17 13:48:13.087322] 2022-08-17T13:48:13.0878188Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:48:13.087405] 2022-08-17T13:48:14.4347400Z CUDA not available, skipping tests 2022-08-17T13:48:14.4360659Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl 2022-08-17T13:48:14.4372224Z 2022-08-17T13:48:14.4372633Z Running tests... 2022-08-17T13:48:14.4373280Z ---------------------------------------------------------------------- 2022-08-17T13:48:14.4373599Z 2022-08-17T13:48:14.4373977Z ---------------------------------------------------------------------- 2022-08-17T13:48:14.4374449Z Ran 0 tests in 0.000s 2022-08-17T13:48:14.4374661Z 2022-08-17T13:48:14.4374770Z OK 2022-08-17T13:48:14.4374944Z 2022-08-17T13:48:14.4375110Z Generating XML reports... 2022-08-17T13:48:14.7316846Z Running distributed/test_pg_wrapper ... [2022-08-17 13:48:14.731240] 2022-08-17T13:48:14.7317464Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:48:14.731331] 2022-08-17T13:48:15.8423857Z 2022-08-17T13:48:15.8424400Z 2022-08-17T13:48:15.8425790Z , <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-08-17T13:48:15.8427230Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8427566Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8427909Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8428282Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8428661Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8429009Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8429337Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8429699Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8430067Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-08-17T13:48:15.8430887Z , <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-08-17T13:48:15.8431636Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) 2022-08-17T13:48:15.8431966Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-08-17T13:48:15.8432320Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-08-17T13:48:15.8432669Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-08-17T13:48:15.8433086Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-08-17T13:48:16.8082095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:16.8082512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:16.8084471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:16.8084847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:16.9649827Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:16.9659729Z 2022-08-17T13:48:16.9659873Z Running tests... 2022-08-17T13:48:16.9660429Z ---------------------------------------------------------------------- 2022-08-17T13:48:17.2349939Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29648 2022-08-17T13:48:17.2378781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29649 2022-08-17T13:48:17.2407420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29650 2022-08-17T13:48:17.2439500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29651 2022-08-17T13:48:18.3432364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:18.3433017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:18.3434436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:18.3435318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:18.3481766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:18.3482420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:18.3485032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:18.3485659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:18.3738839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:18.3739516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:18.3741527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:18.3742149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:18.3749862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:18.3750743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:18.3752403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:18.3753215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:18.5010192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:18.5064677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:18.5324578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:18.5331698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:18.5474084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:18.5574937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:18.5675718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:48:18.5676353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:18.5677229Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:18.5678010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:18.5678574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:18.5679110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:18.6504454Z [E ProcessGroupGloo.cpp:2798] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:48:18.6504992Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2022-08-17T13:48:18.6603890Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:48:18.6704455Z [E ProcessGroupGloo.cpp:136] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-08-17T13:48:19.0503377Z ok (2.082s) 2022-08-17T13:48:19.0510223Z 2022-08-17T13:48:19.0511190Z ---------------------------------------------------------------------- 2022-08-17T13:48:19.0511658Z Ran 1 test in 2.083s 2022-08-17T13:48:19.0511863Z 2022-08-17T13:48:19.0511979Z OK 2022-08-17T13:48:19.0512430Z 2022-08-17T13:48:19.0512592Z Generating XML reports... 2022-08-17T13:48:19.0580446Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134816.xml 2022-08-17T13:48:20.3541778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:20.3542235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:20.3543513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:20.3544047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:20.5054241Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:20.5063695Z 2022-08-17T13:48:20.5063830Z Running tests... 2022-08-17T13:48:20.5064591Z ---------------------------------------------------------------------- 2022-08-17T13:48:20.7688368Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29739 2022-08-17T13:48:20.7714272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29740 2022-08-17T13:48:20.7742928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29741 2022-08-17T13:48:20.7773747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29742 2022-08-17T13:48:21.8745267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:21.8745901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:21.8748013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:21.8749257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:21.9129735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:21.9130725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:21.9131563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:21.9132202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:21.9589533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:21.9590195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:21.9591612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:21.9592272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:21.9709919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:21.9710591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:21.9712662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:21.9713311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:22.0323508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:22.0703280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:22.1164115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:22.1256901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:22.1436556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:22.1576590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:22.1577456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:48:22.1578700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:22.1579842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:22.1641014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:22.1679498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:22.1680432Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:22.6823280Z ok (2.176s) 2022-08-17T13:48:22.6823467Z 2022-08-17T13:48:22.6823999Z ---------------------------------------------------------------------- 2022-08-17T13:48:22.6824675Z Ran 1 test in 2.176s 2022-08-17T13:48:22.6824801Z 2022-08-17T13:48:22.6824851Z OK 2022-08-17T13:48:22.6824950Z 2022-08-17T13:48:22.6825049Z Generating XML reports... 2022-08-17T13:48:22.6872882Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134820.xml 2022-08-17T13:48:23.9763484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:23.9763859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:23.9766038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:23.9766424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:24.1307475Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:24.1317592Z 2022-08-17T13:48:24.1317978Z Running tests... 2022-08-17T13:48:24.1318392Z ---------------------------------------------------------------------- 2022-08-17T13:48:24.4004048Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29830 2022-08-17T13:48:24.4031500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29831 2022-08-17T13:48:24.4059893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29832 2022-08-17T13:48:24.4091872Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29833 2022-08-17T13:48:25.4348417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:25.4349043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:25.4350515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:25.4351148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:25.4494723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:25.4495405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:25.4497381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:25.4498041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:25.4755668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:25.4756285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:25.4757814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:25.4758878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:25.4807424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:25.4807948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:25.4810986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:25.4811529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:25.5930167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:25.6066519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:25.6357533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:25.6380192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:25.9135317Z skip: Need at least 4 CUDA devices (1.781s) 2022-08-17T13:48:25.9135507Z 2022-08-17T13:48:25.9135827Z ---------------------------------------------------------------------- 2022-08-17T13:48:25.9136102Z Ran 1 test in 1.782s 2022-08-17T13:48:25.9136226Z 2022-08-17T13:48:25.9136304Z OK (skipped=1) 2022-08-17T13:48:25.9136417Z 2022-08-17T13:48:25.9136506Z Generating XML reports... 2022-08-17T13:48:25.9185182Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134824.xml 2022-08-17T13:48:27.2004720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:27.2005232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:27.2006529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:27.2007071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:27.3565283Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:27.3574251Z 2022-08-17T13:48:27.3574390Z Running tests... 2022-08-17T13:48:27.3575050Z ---------------------------------------------------------------------- 2022-08-17T13:48:27.6228520Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29885 2022-08-17T13:48:27.6256798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29886 2022-08-17T13:48:27.6284939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29887 2022-08-17T13:48:27.6315274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29888 2022-08-17T13:48:28.6300653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:28.6301346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:28.6302749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:28.6303403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:28.6320069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:28.6320727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:28.6324423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:28.6325402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:28.6840648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:28.6841593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:28.6842345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:28.6842849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:28.7036979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:28.7037734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:28.7039207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:28.7039951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:28.7876235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:28.7904204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:28.8434659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:28.8632602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:29.1357622Z skip: Need at least 4 CUDA devices (1.778s) 2022-08-17T13:48:29.1357873Z 2022-08-17T13:48:29.1358191Z ---------------------------------------------------------------------- 2022-08-17T13:48:29.1358452Z Ran 1 test in 1.778s 2022-08-17T13:48:29.1358572Z 2022-08-17T13:48:29.1358655Z OK (skipped=1) 2022-08-17T13:48:29.1358770Z 2022-08-17T13:48:29.1358866Z Generating XML reports... 2022-08-17T13:48:29.1406131Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134827.xml 2022-08-17T13:48:30.4470712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:30.4471130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:30.4472737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:30.4473112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:30.5988329Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:30.5998232Z 2022-08-17T13:48:30.5998319Z Running tests... 2022-08-17T13:48:30.5998876Z ---------------------------------------------------------------------- 2022-08-17T13:48:30.8668084Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29940 2022-08-17T13:48:30.8696255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29941 2022-08-17T13:48:30.8724250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29942 2022-08-17T13:48:30.8756179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29943 2022-08-17T13:48:31.9266853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:31.9267502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:31.9270903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:31.9271520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:31.9453841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:31.9454530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:31.9457322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:31.9458281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:31.9558031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:31.9558686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:31.9561526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:31.9562145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:31.9729800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:31.9730474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:31.9732400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:31.9733054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:32.0854584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:32.1053856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:32.1150574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:32.1329760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:32.1666630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:32.1767724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:48:32.1768347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:32.1768927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:32.1769959Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:32.1770918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:32.1771876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:32.1772708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:32.2385197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:48:32.2485111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:48:32.2587896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-08-17T13:48:32.2588596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:48:32.2589455Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:32.2590238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:32.2591035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:32.2591645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:32.6803935Z ok (2.080s) 2022-08-17T13:48:32.6804172Z 2022-08-17T13:48:32.6804708Z ---------------------------------------------------------------------- 2022-08-17T13:48:32.6805088Z Ran 1 test in 2.080s 2022-08-17T13:48:32.6805209Z 2022-08-17T13:48:32.6805273Z OK 2022-08-17T13:48:32.6805604Z 2022-08-17T13:48:32.6805686Z Generating XML reports... 2022-08-17T13:48:32.6852584Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134830.xml 2022-08-17T13:48:33.9935281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:33.9935705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:33.9936474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:33.9936907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:34.1473077Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:34.1483754Z 2022-08-17T13:48:34.1483866Z Running tests... 2022-08-17T13:48:34.1484377Z ---------------------------------------------------------------------- 2022-08-17T13:48:34.4160121Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30043 2022-08-17T13:48:34.4188906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30044 2022-08-17T13:48:34.4217725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30045 2022-08-17T13:48:34.4249840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30046 2022-08-17T13:48:35.5106823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:35.5107495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:35.5108239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:35.5108842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:35.5281638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:35.5282338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:35.5284060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:35.5284706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:35.5394839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:35.5395519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:35.5397713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:35.5398373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:35.5638879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:35.5639474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:35.5641425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:35.5641960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:35.6873087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:35.6888405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:35.7008066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:35.7244894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:35.7385490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:35.7486583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:35.7588167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:48:35.7589156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:35.7590256Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:35.7590821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:35.7591358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:35.7591902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:36.3300203Z ok (2.181s) 2022-08-17T13:48:36.3300503Z 2022-08-17T13:48:36.3300883Z ---------------------------------------------------------------------- 2022-08-17T13:48:36.3301405Z Ran 1 test in 2.181s 2022-08-17T13:48:36.3301532Z 2022-08-17T13:48:36.3301596Z OK 2022-08-17T13:48:36.3301695Z 2022-08-17T13:48:36.3301791Z Generating XML reports... 2022-08-17T13:48:36.3348046Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134834.xml 2022-08-17T13:48:37.6208893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:37.6209291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:37.6210863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:37.6211242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:37.7766344Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:37.7776703Z 2022-08-17T13:48:37.7776850Z Running tests... 2022-08-17T13:48:37.7777526Z ---------------------------------------------------------------------- 2022-08-17T13:48:38.0419078Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30134 2022-08-17T13:48:38.0445787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30135 2022-08-17T13:48:38.0473879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30136 2022-08-17T13:48:38.0507196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30137 2022-08-17T13:48:39.1190418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:39.1191089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:39.1192728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:39.1193389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:39.1245570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:39.1246389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:39.1249349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:39.1249955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:39.1409419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:39.1410080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:39.1413008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:39.1414005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:39.1485458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:39.1486294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:39.1488443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:39.1489075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:39.2768950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:39.2841573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:39.2975656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:39.3052468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:39.6552059Z skip: Need at least 4 CUDA devices (1.877s) 2022-08-17T13:48:39.6552402Z 2022-08-17T13:48:39.6552761Z ---------------------------------------------------------------------- 2022-08-17T13:48:39.6553024Z Ran 1 test in 1.877s 2022-08-17T13:48:39.6553146Z 2022-08-17T13:48:39.6553221Z OK (skipped=1) 2022-08-17T13:48:39.6553336Z 2022-08-17T13:48:39.6553427Z Generating XML reports... 2022-08-17T13:48:39.6601058Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134837.xml 2022-08-17T13:48:40.9390358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:40.9390794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:40.9392333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:40.9392729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:41.0939682Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:41.0949268Z 2022-08-17T13:48:41.0949467Z Running tests... 2022-08-17T13:48:41.0949813Z ---------------------------------------------------------------------- 2022-08-17T13:48:41.3602458Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30189 2022-08-17T13:48:41.3631597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30190 2022-08-17T13:48:41.3658616Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30191 2022-08-17T13:48:41.3690588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30192 2022-08-17T13:48:42.4572303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:42.4572976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:42.4576366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:42.4577027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:42.4675024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:42.4675681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:42.4678379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:42.4678975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:42.4856540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:42.4857488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:42.4858890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:42.4859541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:42.4920657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:42.4921306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:42.4923887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:42.4924618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:42.6167782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:42.6261430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:42.6509877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:42.6523440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:42.9736527Z skip: Need at least 4 CUDA devices (1.878s) 2022-08-17T13:48:42.9736808Z 2022-08-17T13:48:42.9737366Z ---------------------------------------------------------------------- 2022-08-17T13:48:42.9737674Z Ran 1 test in 1.879s 2022-08-17T13:48:42.9737797Z 2022-08-17T13:48:42.9737874Z OK (skipped=1) 2022-08-17T13:48:42.9737987Z 2022-08-17T13:48:42.9738064Z Generating XML reports... 2022-08-17T13:48:42.9785086Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134841.xml 2022-08-17T13:48:44.2950283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:44.2950687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:44.2951928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:44.2952439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:44.4487733Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:44.4497236Z 2022-08-17T13:48:44.4497589Z Running tests... 2022-08-17T13:48:44.4498023Z ---------------------------------------------------------------------- 2022-08-17T13:48:44.7139464Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30244 2022-08-17T13:48:44.7166104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30245 2022-08-17T13:48:44.7194012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30246 2022-08-17T13:48:44.7225270Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30247 2022-08-17T13:48:45.7655004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:45.7655673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:45.7656552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:45.7657186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:45.7787356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:45.7788030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:45.7789742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:45.7790621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:45.8179842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:45.8180566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:45.8182336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:45.8183040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:45.8303413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:45.8303983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:45.8306038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:45.8306586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:45.9221418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-08-17T13:48:45.9351013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-08-17T13:48:45.9761865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-08-17T13:48:45.9857241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-08-17T13:48:46.0276298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:48:46.0478653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-08-17T13:48:46.0579777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-08-17T13:48:46.0592679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-08-17T13:48:46.0593926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:46.0683757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:46.0684415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:46.0685029Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-08-17T13:48:46.1297435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-08-17T13:48:46.1398373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-08-17T13:48:46.1500000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-08-17T13:48:46.1500932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-08-17T13:48:46.1502049Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:46.1503035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:46.1504019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:46.1504683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-08-17T13:48:46.6274066Z ok (2.177s) 2022-08-17T13:48:46.6274313Z 2022-08-17T13:48:46.6274785Z ---------------------------------------------------------------------- 2022-08-17T13:48:46.6275218Z Ran 1 test in 2.178s 2022-08-17T13:48:46.6275411Z 2022-08-17T13:48:46.6275492Z OK 2022-08-17T13:48:46.6275937Z 2022-08-17T13:48:46.6276093Z Generating XML reports... 2022-08-17T13:48:46.6323913Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134844.xml 2022-08-17T13:48:47.9250085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:47.9250734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:47.9251877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:47.9252468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:48.0792250Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:48.0802386Z 2022-08-17T13:48:48.0802686Z Running tests... 2022-08-17T13:48:48.0803342Z ---------------------------------------------------------------------- 2022-08-17T13:48:48.0808663Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:48:48.0809467Z 2022-08-17T13:48:48.0809735Z ---------------------------------------------------------------------- 2022-08-17T13:48:48.0810000Z Ran 1 test in 0.001s 2022-08-17T13:48:48.0810108Z 2022-08-17T13:48:48.0810188Z OK (skipped=1) 2022-08-17T13:48:48.0810304Z 2022-08-17T13:48:48.0810396Z Generating XML reports... 2022-08-17T13:48:48.0834254Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134848.xml 2022-08-17T13:48:49.2398406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:49.2398788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:49.2400055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:49.2400446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:49.3921930Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:49.3930963Z 2022-08-17T13:48:49.3931247Z Running tests... 2022-08-17T13:48:49.3931634Z ---------------------------------------------------------------------- 2022-08-17T13:48:49.3936009Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:48:49.3936331Z 2022-08-17T13:48:49.3936966Z ---------------------------------------------------------------------- 2022-08-17T13:48:49.3937341Z Ran 1 test in 0.001s 2022-08-17T13:48:49.3937469Z 2022-08-17T13:48:49.3937557Z OK (skipped=1) 2022-08-17T13:48:49.3937673Z 2022-08-17T13:48:49.3937764Z Generating XML reports... 2022-08-17T13:48:49.3963163Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134849.xml 2022-08-17T13:48:50.5637855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:50.5638247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:50.5640142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:50.5640586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:50.7148647Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:50.7158658Z 2022-08-17T13:48:50.7158752Z Running tests... 2022-08-17T13:48:50.7159646Z ---------------------------------------------------------------------- 2022-08-17T13:48:50.7164565Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:48:50.7165405Z 2022-08-17T13:48:50.7165661Z ---------------------------------------------------------------------- 2022-08-17T13:48:50.7166102Z Ran 1 test in 0.001s 2022-08-17T13:48:50.7166231Z 2022-08-17T13:48:50.7166311Z OK (skipped=1) 2022-08-17T13:48:50.7166424Z 2022-08-17T13:48:50.7166514Z Generating XML reports... 2022-08-17T13:48:50.7191194Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134850.xml 2022-08-17T13:48:51.8620469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:51.8620857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:51.8622557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:51.8622936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:52.0125446Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:52.0134681Z 2022-08-17T13:48:52.0134879Z Running tests... 2022-08-17T13:48:52.0135517Z ---------------------------------------------------------------------- 2022-08-17T13:48:52.0140440Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:48:52.0140984Z 2022-08-17T13:48:52.0141370Z ---------------------------------------------------------------------- 2022-08-17T13:48:52.0141628Z Ran 1 test in 0.001s 2022-08-17T13:48:52.0141750Z 2022-08-17T13:48:52.0141828Z OK (skipped=1) 2022-08-17T13:48:52.0141944Z 2022-08-17T13:48:52.0142020Z Generating XML reports... 2022-08-17T13:48:52.0166909Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134852.xml 2022-08-17T13:48:53.1543082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:53.1543517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:53.1544769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:53.1545147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:53.3068844Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-08-17T13:48:53.3078375Z 2022-08-17T13:48:53.3078662Z Running tests... 2022-08-17T13:48:53.3079070Z ---------------------------------------------------------------------- 2022-08-17T13:48:53.3084202Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-08-17T13:48:53.3084723Z 2022-08-17T13:48:53.3085070Z ---------------------------------------------------------------------- 2022-08-17T13:48:53.3085341Z Ran 1 test in 0.001s 2022-08-17T13:48:53.3085480Z 2022-08-17T13:48:53.3085542Z OK (skipped=1) 2022-08-17T13:48:53.3085658Z 2022-08-17T13:48:53.3085749Z Generating XML reports... 2022-08-17T13:48:53.3110803Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134853.xml 2022-08-17T13:48:53.7528828Z Running distributed/test_store ... [2022-08-17 13:48:53.752462] 2022-08-17T13:48:53.7529427Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-08-17 13:48:53.752546] 2022-08-17T13:48:54.8471470Z test_compare_set (__main__.FileStoreTest) 2022-08-17T13:48:54.8471810Z test_set_get (__main__.FileStoreTest) 2022-08-17T13:48:54.8472166Z test_compare_set (__main__.HashStoreTest) 2022-08-17T13:48:54.8472402Z test_set_get (__main__.HashStoreTest) 2022-08-17T13:48:54.8472692Z test_compare_set (__main__.PrefixFileStoreTest) 2022-08-17T13:48:54.8473996Z test_set_get (__main__.PrefixFileStoreTest) 2022-08-17T13:48:54.8474763Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-08-17T13:48:54.8475023Z test_set_get (__main__.PrefixTCPStoreTest) 2022-08-17T13:48:54.8475265Z test_set_get (__main__.PythonStoreTest) 2022-08-17T13:48:54.8475518Z test_nominal (__main__.RendezvousEnvTest) 2022-08-17T13:48:54.8475769Z test_common_errors (__main__.RendezvousFileTest) 2022-08-17T13:48:54.8477929Z test_nominal (__main__.RendezvousFileTest) 2022-08-17T13:48:54.8478356Z test_common_errors (__main__.RendezvousTCPTest) 2022-08-17T13:48:54.8478689Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-08-17T13:48:54.8478935Z test_nominal (__main__.RendezvousTCPTest) 2022-08-17T13:48:54.8479222Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-08-17T13:48:54.8479491Z test_unknown_handler (__main__.RendezvousTest) 2022-08-17T13:48:54.8479747Z test_url_with_node_params (__main__.RendezvousTest) 2022-08-17T13:48:54.8480067Z test_address_already_in_use (__main__.TCPStoreTest) 2022-08-17T13:48:54.8480314Z test_compare_set (__main__.TCPStoreTest) 2022-08-17T13:48:54.8480638Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-08-17T13:48:54.8481085Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-08-17T13:48:54.8481446Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-08-17T13:48:54.8481717Z test_multitenancy (__main__.TCPStoreTest) 2022-08-17T13:48:54.8481967Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-08-17T13:48:54.8482249Z test_set_get (__main__.TCPStoreTest) 2022-08-17T13:48:55.8003210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:55.8003611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:55.8006590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:55.8007163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:55.9557642Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:48:55.9567056Z 2022-08-17T13:48:55.9567211Z Running tests... 2022-08-17T13:48:55.9567868Z ---------------------------------------------------------------------- 2022-08-17T13:48:56.2205069Z test_compare_set (__main__.FileStoreTest) ... ok (0.263s) 2022-08-17T13:48:56.2205345Z 2022-08-17T13:48:56.2205730Z ---------------------------------------------------------------------- 2022-08-17T13:48:56.2206216Z Ran 1 test in 0.264s 2022-08-17T13:48:56.2206337Z 2022-08-17T13:48:56.2206406Z OK 2022-08-17T13:48:56.2206503Z 2022-08-17T13:48:56.2206581Z Generating XML reports... 2022-08-17T13:48:56.2232807Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817134855.xml 2022-08-17T13:48:57.4728794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:57.4729177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:57.4732068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:57.4732449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:57.6266147Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:48:57.6276455Z 2022-08-17T13:48:57.6276880Z Running tests... 2022-08-17T13:48:57.6277306Z ---------------------------------------------------------------------- 2022-08-17T13:48:57.8856820Z test_set_get (__main__.FileStoreTest) ... ok (0.258s) 2022-08-17T13:48:57.8857100Z 2022-08-17T13:48:57.8857614Z ---------------------------------------------------------------------- 2022-08-17T13:48:57.8857967Z Ran 1 test in 0.258s 2022-08-17T13:48:57.8858073Z 2022-08-17T13:48:57.8858137Z OK 2022-08-17T13:48:57.8858232Z 2022-08-17T13:48:57.8858330Z Generating XML reports... 2022-08-17T13:48:57.8885350Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817134857.xml 2022-08-17T13:48:59.1331709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:48:59.1332091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:48:59.1334704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:48:59.1335084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:48:59.2877494Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:48:59.2886802Z 2022-08-17T13:48:59.2886936Z Running tests... 2022-08-17T13:48:59.2887315Z ---------------------------------------------------------------------- 2022-08-17T13:48:59.5536715Z test_compare_set (__main__.HashStoreTest) ... ok (0.265s) 2022-08-17T13:48:59.5536982Z 2022-08-17T13:48:59.5537534Z ---------------------------------------------------------------------- 2022-08-17T13:48:59.5537905Z Ran 1 test in 0.265s 2022-08-17T13:48:59.5538026Z 2022-08-17T13:48:59.5538289Z OK 2022-08-17T13:48:59.5538391Z 2022-08-17T13:48:59.5538481Z Generating XML reports... 2022-08-17T13:48:59.5565195Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817134859.xml 2022-08-17T13:49:00.7901916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:00.7902402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:00.7904708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:00.7905168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:00.9465100Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:00.9476287Z 2022-08-17T13:49:00.9476435Z Running tests... 2022-08-17T13:49:00.9477116Z ---------------------------------------------------------------------- 2022-08-17T13:49:01.2099093Z test_set_get (__main__.HashStoreTest) ... ok (0.262s) 2022-08-17T13:49:01.2099422Z 2022-08-17T13:49:01.2099832Z ---------------------------------------------------------------------- 2022-08-17T13:49:01.2100078Z Ran 1 test in 0.262s 2022-08-17T13:49:01.2100201Z 2022-08-17T13:49:01.2100266Z OK 2022-08-17T13:49:01.2100362Z 2022-08-17T13:49:01.2100457Z Generating XML reports... 2022-08-17T13:49:01.2125792Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817134900.xml 2022-08-17T13:49:02.4651812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:02.4652208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:02.4655033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:02.4655422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:02.6159358Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:02.6169234Z 2022-08-17T13:49:02.6169340Z Running tests... 2022-08-17T13:49:02.6169829Z ---------------------------------------------------------------------- 2022-08-17T13:49:02.8731997Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.256s) 2022-08-17T13:49:02.8732396Z 2022-08-17T13:49:02.8732850Z ---------------------------------------------------------------------- 2022-08-17T13:49:02.8733113Z Ran 1 test in 0.256s 2022-08-17T13:49:02.8733234Z 2022-08-17T13:49:02.8733301Z OK 2022-08-17T13:49:02.8733386Z 2022-08-17T13:49:02.8733476Z Generating XML reports... 2022-08-17T13:49:02.8757108Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817134902.xml 2022-08-17T13:49:04.1092342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:04.1092746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:04.1094463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:04.1094829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:04.2635747Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:04.2645458Z 2022-08-17T13:49:04.2645609Z Running tests... 2022-08-17T13:49:04.2646247Z ---------------------------------------------------------------------- 2022-08-17T13:49:04.5221924Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.257s) 2022-08-17T13:49:04.5222158Z 2022-08-17T13:49:04.5222687Z ---------------------------------------------------------------------- 2022-08-17T13:49:04.5223183Z Ran 1 test in 0.257s 2022-08-17T13:49:04.5223384Z 2022-08-17T13:49:04.5223469Z OK 2022-08-17T13:49:04.5223571Z 2022-08-17T13:49:04.5223824Z Generating XML reports... 2022-08-17T13:49:04.5248057Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817134904.xml 2022-08-17T13:49:05.7711321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:05.7711747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:05.7714396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:05.7714897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:05.9261167Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:05.9271146Z 2022-08-17T13:49:05.9271442Z Running tests... 2022-08-17T13:49:05.9272081Z ---------------------------------------------------------------------- 2022-08-17T13:49:06.1858790Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.259s) 2022-08-17T13:49:06.1859163Z 2022-08-17T13:49:06.1859629Z ---------------------------------------------------------------------- 2022-08-17T13:49:06.1859899Z Ran 1 test in 0.259s 2022-08-17T13:49:06.1860021Z 2022-08-17T13:49:06.1860087Z OK 2022-08-17T13:49:06.1860169Z 2022-08-17T13:49:06.1860259Z Generating XML reports... 2022-08-17T13:49:06.1885515Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817134905.xml 2022-08-17T13:49:07.4386092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:07.4386487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:07.4389313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:07.4389703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:07.5900902Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:07.5911444Z 2022-08-17T13:49:07.5911760Z Running tests... 2022-08-17T13:49:07.5912150Z ---------------------------------------------------------------------- 2022-08-17T13:49:07.8468325Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.255s) 2022-08-17T13:49:07.8468662Z 2022-08-17T13:49:07.8469150Z ---------------------------------------------------------------------- 2022-08-17T13:49:07.8469642Z Ran 1 test in 0.256s 2022-08-17T13:49:07.8469843Z 2022-08-17T13:49:07.8469951Z OK 2022-08-17T13:49:07.8470258Z 2022-08-17T13:49:07.8470413Z Generating XML reports... 2022-08-17T13:49:07.8495939Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817134907.xml 2022-08-17T13:49:09.0730399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:09.0731375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:09.0733429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:09.0734061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:09.2295462Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:09.2305768Z 2022-08-17T13:49:09.2306060Z Running tests... 2022-08-17T13:49:09.2306705Z ---------------------------------------------------------------------- 2022-08-17T13:49:09.4917066Z test_set_get (__main__.PythonStoreTest) ... ok (0.261s) 2022-08-17T13:49:09.4917384Z 2022-08-17T13:49:09.4917881Z ---------------------------------------------------------------------- 2022-08-17T13:49:09.4918362Z Ran 1 test in 0.261s 2022-08-17T13:49:09.4918578Z 2022-08-17T13:49:09.4918693Z OK 2022-08-17T13:49:09.4918867Z 2022-08-17T13:49:09.4919032Z Generating XML reports... 2022-08-17T13:49:09.4945383Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220817134909.xml 2022-08-17T13:49:10.7517601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:10.7518268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:10.7521633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:10.7522236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:10.9077206Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:10.9087169Z 2022-08-17T13:49:10.9087656Z Running tests... 2022-08-17T13:49:10.9088080Z ---------------------------------------------------------------------- 2022-08-17T13:49:11.1679320Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.259s) 2022-08-17T13:49:11.1679646Z 2022-08-17T13:49:11.1679979Z ---------------------------------------------------------------------- 2022-08-17T13:49:11.1680241Z Ran 1 test in 0.259s 2022-08-17T13:49:11.1680361Z 2022-08-17T13:49:11.1680425Z OK 2022-08-17T13:49:11.1680507Z 2022-08-17T13:49:11.1680602Z Generating XML reports... 2022-08-17T13:49:11.1706140Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220817134910.xml 2022-08-17T13:49:12.4187160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:12.4187807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:12.4190518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:12.4191124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:12.5729256Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:12.5739631Z 2022-08-17T13:49:12.5740074Z Running tests... 2022-08-17T13:49:12.5740514Z ---------------------------------------------------------------------- 2022-08-17T13:49:12.8327701Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.259s) 2022-08-17T13:49:12.8327987Z 2022-08-17T13:49:12.8328317Z ---------------------------------------------------------------------- 2022-08-17T13:49:12.8328576Z Ran 1 test in 0.259s 2022-08-17T13:49:12.8328696Z 2022-08-17T13:49:12.8328762Z OK 2022-08-17T13:49:12.8328856Z 2022-08-17T13:49:12.8328950Z Generating XML reports... 2022-08-17T13:49:12.8353030Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817134912.xml 2022-08-17T13:49:14.0715440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:14.0716102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:14.0718818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:14.0719322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:14.2279910Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:14.2289829Z 2022-08-17T13:49:14.2289969Z Running tests... 2022-08-17T13:49:14.2290611Z ---------------------------------------------------------------------- 2022-08-17T13:49:14.4895623Z test_nominal (__main__.RendezvousFileTest) ... ok (0.260s) 2022-08-17T13:49:14.4895934Z 2022-08-17T13:49:14.4896453Z ---------------------------------------------------------------------- 2022-08-17T13:49:14.4896846Z Ran 1 test in 0.260s 2022-08-17T13:49:14.4896954Z 2022-08-17T13:49:14.4897017Z OK 2022-08-17T13:49:14.4897115Z 2022-08-17T13:49:14.4897204Z Generating XML reports... 2022-08-17T13:49:14.4921397Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817134914.xml 2022-08-17T13:49:15.7376596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:15.7377002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:15.7379654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:15.7380163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:15.8949100Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:15.8958456Z 2022-08-17T13:49:15.8958614Z Running tests... 2022-08-17T13:49:15.8959621Z ---------------------------------------------------------------------- 2022-08-17T13:49:16.1545617Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.258s) 2022-08-17T13:49:16.1545911Z 2022-08-17T13:49:16.1546266Z ---------------------------------------------------------------------- 2022-08-17T13:49:16.1546539Z Ran 1 test in 0.259s 2022-08-17T13:49:16.1546659Z 2022-08-17T13:49:16.1546724Z OK 2022-08-17T13:49:16.1546806Z 2022-08-17T13:49:16.1546927Z Generating XML reports... 2022-08-17T13:49:16.1571502Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134915.xml 2022-08-17T13:49:17.3945552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:17.3945953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:17.3948561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:17.3948940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:17.5493173Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:17.5502544Z 2022-08-17T13:49:17.5502691Z Running tests... 2022-08-17T13:49:17.5503834Z ---------------------------------------------------------------------- 2022-08-17T13:49:17.8345210Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known). 2022-08-17T13:49:17.8345780Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-08-17T13:49:17.8348198Z ok (0.284s) 2022-08-17T13:49:17.8348418Z 2022-08-17T13:49:17.8348882Z ---------------------------------------------------------------------- 2022-08-17T13:49:17.8349358Z Ran 1 test in 0.285s 2022-08-17T13:49:17.8349495Z 2022-08-17T13:49:17.8349546Z OK 2022-08-17T13:49:17.8349641Z 2022-08-17T13:49:17.8349760Z Generating XML reports... 2022-08-17T13:49:17.8377718Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134917.xml 2022-08-17T13:49:19.0757995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:19.0758398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:19.0760791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:19.0761314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:19.2312429Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:19.2321966Z 2022-08-17T13:49:19.2322092Z Running tests... 2022-08-17T13:49:19.2322697Z ---------------------------------------------------------------------- 2022-08-17T13:49:19.4944206Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.262s) 2022-08-17T13:49:19.4944490Z 2022-08-17T13:49:19.4944798Z ---------------------------------------------------------------------- 2022-08-17T13:49:19.4945075Z Ran 1 test in 0.262s 2022-08-17T13:49:19.4945196Z 2022-08-17T13:49:19.4945460Z OK 2022-08-17T13:49:19.4945546Z 2022-08-17T13:49:19.4945635Z Generating XML reports... 2022-08-17T13:49:19.4971762Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134919.xml 2022-08-17T13:49:20.7352991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:20.7353375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:20.7355482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:20.7355981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:20.8900881Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:20.8910794Z 2022-08-17T13:49:20.8911098Z Running tests... 2022-08-17T13:49:20.8911480Z ---------------------------------------------------------------------- 2022-08-17T13:49:31.3564859Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.465s) 2022-08-17T13:49:31.3565077Z 2022-08-17T13:49:31.3565386Z ---------------------------------------------------------------------- 2022-08-17T13:49:31.3565645Z Ran 1 test in 10.465s 2022-08-17T13:49:31.3565766Z 2022-08-17T13:49:31.3565829Z OK 2022-08-17T13:49:31.3566203Z 2022-08-17T13:49:31.3566295Z Generating XML reports... 2022-08-17T13:49:31.3595306Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134920.xml 2022-08-17T13:49:32.6294150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:32.6294874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:32.6297497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:32.6297926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:32.7832347Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:32.7841677Z 2022-08-17T13:49:32.7841961Z Running tests... 2022-08-17T13:49:32.7842405Z ---------------------------------------------------------------------- 2022-08-17T13:49:33.0384910Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.254s) 2022-08-17T13:49:33.0385099Z 2022-08-17T13:49:33.0385490Z ---------------------------------------------------------------------- 2022-08-17T13:49:33.0385760Z Ran 1 test in 0.254s 2022-08-17T13:49:33.0385885Z 2022-08-17T13:49:33.0385949Z OK 2022-08-17T13:49:33.0386046Z 2022-08-17T13:49:33.0386130Z Generating XML reports... 2022-08-17T13:49:33.0410807Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817134932.xml 2022-08-17T13:49:34.2903107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:34.2903501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:34.2906107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:34.2906492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:34.4422659Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:34.4431843Z 2022-08-17T13:49:34.4432329Z Running tests... 2022-08-17T13:49:34.4432749Z ---------------------------------------------------------------------- 2022-08-17T13:49:34.7005664Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.257s) 2022-08-17T13:49:34.7006113Z 2022-08-17T13:49:34.7006650Z ---------------------------------------------------------------------- 2022-08-17T13:49:34.7007002Z Ran 1 test in 0.257s 2022-08-17T13:49:34.7007122Z 2022-08-17T13:49:34.7007185Z OK 2022-08-17T13:49:34.7007281Z 2022-08-17T13:49:34.7007604Z Generating XML reports... 2022-08-17T13:49:34.7031279Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817134934.xml 2022-08-17T13:49:35.9531884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:35.9532314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:35.9535147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:35.9535605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:36.1076596Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:36.1086971Z 2022-08-17T13:49:36.1087424Z Running tests... 2022-08-17T13:49:36.1087860Z ---------------------------------------------------------------------- 2022-08-17T13:49:36.3663237Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:45539 (errno: 98 - Address already in use). 2022-08-17T13:49:36.3675017Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:45539 (errno: 98 - Address already in use). 2022-08-17T13:49:36.3675419Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-08-17T13:49:36.3679634Z ok (0.259s) 2022-08-17T13:49:36.3679778Z 2022-08-17T13:49:36.3680175Z ---------------------------------------------------------------------- 2022-08-17T13:49:36.3680658Z Ran 1 test in 0.259s 2022-08-17T13:49:36.3680873Z 2022-08-17T13:49:36.3680957Z OK 2022-08-17T13:49:36.3681055Z 2022-08-17T13:49:36.3681147Z Generating XML reports... 2022-08-17T13:49:36.3707382Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134936.xml 2022-08-17T13:49:37.6002539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:37.6002949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:37.6005469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:37.6006247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:37.7507552Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:37.7517229Z 2022-08-17T13:49:37.7517533Z Running tests... 2022-08-17T13:49:37.7518150Z ---------------------------------------------------------------------- 2022-08-17T13:49:38.0074368Z test_compare_set (__main__.TCPStoreTest) ... ok (0.256s) 2022-08-17T13:49:38.0074620Z 2022-08-17T13:49:38.0075095Z ---------------------------------------------------------------------- 2022-08-17T13:49:38.0075786Z Ran 1 test in 0.256s 2022-08-17T13:49:38.0075953Z 2022-08-17T13:49:38.0076054Z OK 2022-08-17T13:49:38.0076200Z 2022-08-17T13:49:38.0076351Z Generating XML reports... 2022-08-17T13:49:38.0101953Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134937.xml 2022-08-17T13:49:39.2476803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:39.2477478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:39.2480987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:39.2481578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:39.4041866Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:39.4050682Z 2022-08-17T13:49:39.4051188Z Running tests... 2022-08-17T13:49:39.4051854Z ---------------------------------------------------------------------- 2022-08-17T13:49:39.6631414Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-08-17T13:49:39.6632141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-08-17T13:49:39.7258404Z ok (0.321s) 2022-08-17T13:49:39.7258645Z 2022-08-17T13:49:39.7259217Z ---------------------------------------------------------------------- 2022-08-17T13:49:39.7259579Z Ran 1 test in 0.321s 2022-08-17T13:49:39.7259686Z 2022-08-17T13:49:39.7259750Z OK 2022-08-17T13:49:39.7259846Z 2022-08-17T13:49:39.7259940Z Generating XML reports... 2022-08-17T13:49:39.7286470Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134939.xml 2022-08-17T13:49:40.9642698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:40.9643105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:40.9645601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:40.9646206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:41.1180132Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:41.1190030Z 2022-08-17T13:49:41.1190623Z Running tests... 2022-08-17T13:49:41.1191156Z ---------------------------------------------------------------------- 2022-08-17T13:49:41.3789884Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.260s) 2022-08-17T13:49:41.3790276Z 2022-08-17T13:49:41.3790765Z ---------------------------------------------------------------------- 2022-08-17T13:49:41.3791213Z Ran 1 test in 0.260s 2022-08-17T13:49:41.3791372Z 2022-08-17T13:49:41.3791435Z OK 2022-08-17T13:49:41.3791529Z 2022-08-17T13:49:41.3791619Z Generating XML reports... 2022-08-17T13:49:41.3817063Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134941.xml 2022-08-17T13:49:42.6101077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:42.6101455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:42.6104032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:42.6104557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:42.7651406Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:42.7660686Z 2022-08-17T13:49:42.7660828Z Running tests... 2022-08-17T13:49:42.7661447Z ---------------------------------------------------------------------- 2022-08-17T13:49:43.0239802Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.258s) 2022-08-17T13:49:43.0240050Z 2022-08-17T13:49:43.0240431Z ---------------------------------------------------------------------- 2022-08-17T13:49:43.0240693Z Ran 1 test in 0.258s 2022-08-17T13:49:43.0240813Z 2022-08-17T13:49:43.0240881Z OK 2022-08-17T13:49:43.0240983Z 2022-08-17T13:49:43.0241061Z Generating XML reports... 2022-08-17T13:49:43.0266706Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134942.xml 2022-08-17T13:49:44.2573939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:44.2574346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:44.2577043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:44.2577410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:44.4106657Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:44.4115745Z 2022-08-17T13:49:44.4115889Z Running tests... 2022-08-17T13:49:44.4116380Z ---------------------------------------------------------------------- 2022-08-17T13:49:44.6681233Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.256s) 2022-08-17T13:49:44.6681580Z 2022-08-17T13:49:44.6681952Z ---------------------------------------------------------------------- 2022-08-17T13:49:44.6682203Z Ran 1 test in 0.256s 2022-08-17T13:49:44.6682333Z 2022-08-17T13:49:44.6682399Z OK 2022-08-17T13:49:44.6682497Z 2022-08-17T13:49:44.6682589Z Generating XML reports... 2022-08-17T13:49:44.6707067Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134944.xml 2022-08-17T13:49:45.9143391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:45.9143817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:45.9146077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:45.9146451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:46.0680608Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:46.0690174Z 2022-08-17T13:49:46.0690273Z Running tests... 2022-08-17T13:49:46.0691194Z ---------------------------------------------------------------------- 2022-08-17T13:49:48.3491762Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.280s) 2022-08-17T13:49:48.3492146Z 2022-08-17T13:49:48.3492546Z ---------------------------------------------------------------------- 2022-08-17T13:49:48.3492810Z Ran 1 test in 2.280s 2022-08-17T13:49:48.3492939Z 2022-08-17T13:49:48.3492991Z OK 2022-08-17T13:49:48.3493101Z 2022-08-17T13:49:48.3493194Z Generating XML reports... 2022-08-17T13:49:48.3520549Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134946.xml 2022-08-17T13:49:49.6044812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 39 slow tests 2022-08-17T13:49:49.6045213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-08-17T13:49:49.6047584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 238 disabled tests 2022-08-17T13:49:49.6047967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-08-17T13:49:49.7577249Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-08-17T13:49:49.7586975Z 2022-08-17T13:49:49.7587107Z Running tests... 2022-08-17T13:49:49.7587812Z ---------------------------------------------------------------------- 2022-08-17T13:49:50.0175601Z test_set_get (__main__.TCPStoreTest) ... ok (0.259s) 2022-08-17T13:49:50.0176188Z 2022-08-17T13:49:50.0176597Z ---------------------------------------------------------------------- 2022-08-17T13:49:50.0176870Z Ran 1 test in 0.259s 2022-08-17T13:49:50.0176974Z 2022-08-17T13:49:50.0177039Z OK 2022-08-17T13:49:50.0177136Z 2022-08-17T13:49:50.0177226Z Generating XML reports... 2022-08-17T13:49:50.0204966Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134949.xml 2022-08-17T13:49:50.8485290Z 2022-08-17T13:49:50.8485745Z real 99m7.112s 2022-08-17T13:49:50.8486390Z user 192m27.467s 2022-08-17T13:49:50.8486592Z sys 49m2.963s 2022-08-17T13:49:50.8486764Z + assert_git_not_dirty 2022-08-17T13:49:50.8487211Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-08-17T13:49:50.8487641Z + [[ linux-focal-py3.7-gcc7 != *xla* ]] 2022-08-17T13:49:50.8488008Z ++ git status --porcelain 2022-08-17T13:49:52.3910018Z + git_status= 2022-08-17T13:49:52.3910591Z + [[ -n '' ]] 2022-08-17T13:49:52.3910942Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-08-17T13:49:52.3911166Z + [[ 1 == 1 ]] 2022-08-17T13:49:52.3911341Z + test_rpc 2022-08-17T13:49:52.3911748Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-08-17T13:49:52.3912008Z + echo 'Testing RPC C++ tests' 2022-08-17T13:49:52.3912214Z Testing RPC C++ tests 2022-08-17T13:49:52.3913099Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-08-17T13:49:52.3933359Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-08-17T13:49:52.3941435Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin 2022-08-17T13:49:52.3949455Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc 2022-08-17T13:49:52.3949825Z + mkdir -p test/test-reports/cpp-rpc/test_rpc 2022-08-17T13:49:52.3991634Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml 2022-08-17T13:49:52.6238175Z CUDA not available. Disabling CUDA and MultiCUDA tests 2022-08-17T13:49:52.6238957Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA 2022-08-17T13:49:52.6239286Z [==========] Running 8 tests from 3 test suites. 2022-08-17T13:49:52.6239603Z [----------] Global test environment set-up. 2022-08-17T13:49:52.6239917Z [----------] 4 tests from WireSerialize 2022-08-17T13:49:52.6240200Z [ RUN ] WireSerialize.Base 2022-08-17T13:49:52.6415533Z [ OK ] WireSerialize.Base (17 ms) 2022-08-17T13:49:52.6415983Z [ RUN ] WireSerialize.RecopySparseTensors 2022-08-17T13:49:52.6498122Z [ OK ] WireSerialize.RecopySparseTensors (8 ms) 2022-08-17T13:49:52.6498534Z [ RUN ] WireSerialize.CloneSparseTensors 2022-08-17T13:49:52.6573595Z [ OK ] WireSerialize.CloneSparseTensors (7 ms) 2022-08-17T13:49:52.6573935Z [ RUN ] WireSerialize.Errors 2022-08-17T13:49:52.6598745Z [ OK ] WireSerialize.Errors (2 ms) 2022-08-17T13:49:52.6599246Z [----------] 4 tests from WireSerialize (36 ms total) 2022-08-17T13:49:52.6599505Z 2022-08-17T13:49:52.6599774Z [----------] 1 test from TestE2ETensorPipe 2022-08-17T13:49:52.6600308Z [ RUN ] TestE2ETensorPipe.TestTrainingLoop 2022-08-17T13:49:53.0411778Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-08-17T13:49:53.0422322Z [ OK ] TestE2ETensorPipe.TestTrainingLoop (382 ms) 2022-08-17T13:49:53.0423126Z [----------] 1 test from TestE2ETensorPipe (382 ms total) 2022-08-17T13:49:53.0423735Z 2022-08-17T13:49:53.0424068Z [----------] 3 tests from TensorpipeSerialize 2022-08-17T13:49:53.0424637Z [ RUN ] TensorpipeSerialize.Base 2022-08-17T13:49:53.0425193Z [ OK ] TensorpipeSerialize.Base (0 ms) 2022-08-17T13:49:53.0425847Z [ RUN ] TensorpipeSerialize.RecopySparseTensors 2022-08-17T13:49:53.0497751Z [ OK ] TensorpipeSerialize.RecopySparseTensors (7 ms) 2022-08-17T13:49:53.0498493Z [ RUN ] TensorpipeSerialize.NoDeleterTensors 2022-08-17T13:49:53.0499228Z [ OK ] TensorpipeSerialize.NoDeleterTensors (0 ms) 2022-08-17T13:49:53.0499882Z [----------] 3 tests from TensorpipeSerialize (7 ms total) 2022-08-17T13:49:53.0500695Z 2022-08-17T13:49:53.0501028Z [----------] Global test environment tear-down 2022-08-17T13:49:53.0501776Z [==========] 8 tests from 3 test suites ran. (426 ms total) 2022-08-17T13:49:53.0502265Z [ PASSED ] 8 tests. 2022-08-17T13:49:53.0502500Z 2022-08-17T13:49:53.0502794Z  YOU HAVE 1 DISABLED TEST 2022-08-17T13:49:53.0503159Z 2022-08-17T13:49:53.1156957Z  2022-08-17T13:49:53.1193028Z Prepare all required actions 2022-08-17T13:49:53.1193362Z Getting action download info 2022-08-17T13:49:53.3193617Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-08-17T13:49:53.3193853Z with: 2022-08-17T13:49:53.3194200Z github-token: *** 2022-08-17T13:49:53.3194363Z env: 2022-08-17T13:49:53.3194543Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:53.3194736Z ##[endgroup] 2022-08-17T13:49:53.3221558Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a 2022-08-17T13:49:53.3221793Z with: 2022-08-17T13:49:53.3221964Z shell: bash 2022-08-17T13:49:53.3222150Z timeout_minutes: 10 2022-08-17T13:49:53.3222326Z max_attempts: 5 2022-08-17T13:49:53.3222611Z retry_wait_seconds: 30 2022-08-17T13:49:53.3223021Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}" 2022-08-17T13:49:53.3223464Z polling_interval_seconds: 1 2022-08-17T13:49:53.3223657Z warning_on_retry: true 2022-08-17T13:49:53.3223858Z continue_on_error: false 2022-08-17T13:49:53.3224044Z env: 2022-08-17T13:49:53.3224210Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:53.3224538Z GITHUB_TOKEN: *** 2022-08-17T13:49:53.3224723Z ##[endgroup] 2022-08-17T13:49:53.5681006Z 2022-08-17T13:49:53.5733748Z + python3 -m pip install requests==2.26.0 2022-08-17T13:49:53.7832727Z Defaulting to user installation because normal site-packages is not writeable 2022-08-17T13:49:53.8835268Z Collecting requests==2.26.0 2022-08-17T13:49:53.8978488Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2022-08-17T13:49:53.9533968Z Collecting certifi>=2017.4.17 2022-08-17T13:49:53.9566592Z Downloading certifi-2022.6.15-py3-none-any.whl (160 kB) 2022-08-17T13:49:54.0261417Z Collecting urllib3<1.27,>=1.21.1 2022-08-17T13:49:54.0295329Z Downloading urllib3-1.26.11-py2.py3-none-any.whl (139 kB) 2022-08-17T13:49:54.0870005Z Collecting idna<4,>=2.5; python_version >= "3" 2022-08-17T13:49:54.0900951Z Downloading idna-3.3-py3-none-any.whl (61 kB) 2022-08-17T13:49:54.1445632Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2022-08-17T13:49:54.1476848Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2022-08-17T13:49:54.2076474Z Installing collected packages: certifi, urllib3, idna, charset-normalizer, requests 2022-08-17T13:49:54.3724718Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-08-17T13:49:54.3725271Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-08-17T13:49:54.4150509Z Successfully installed certifi-2022.6.15 charset-normalizer-2.0.12 idna-3.3 requests-2.26.0 urllib3-1.26.11 2022-08-17T13:49:54.4635096Z ++ python3 .github/scripts/get_workflow_job_id.py 2875102080 i-068af22e3918a20a1 2022-08-17T13:49:56.0628842Z + GHA_WORKFLOW_JOB_ID=7878166005 2022-08-17T13:49:56.0629650Z + echo '::set-output name=job-id::7878166005' 2022-08-17T13:49:56.5749034Z Command completed after 1 attempt(s). 2022-08-17T13:49:56.5749324Z 2022-08-17T13:49:56.5862663Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-08-17T13:49:56.5862922Z kill "$MONITOR_SCRIPT_PID" 2022-08-17T13:49:56.5881663Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:49:56.5881888Z env: 2022-08-17T13:49:56.5882073Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:56.5882285Z MONITOR_SCRIPT_PID: 14230 2022-08-17T13:49:56.5882481Z ##[endgroup] 2022-08-17T13:49:56.5962803Z Prepare all required actions 2022-08-17T13:49:56.5963072Z Getting action download info 2022-08-17T13:49:56.7727112Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2) 2022-08-17T13:49:56.9121470Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-08-17T13:49:56.9121704Z with: 2022-08-17T13:49:56.9121934Z file-suffix: test-distributed-1-1-linux.2xlarge_7878166005 2022-08-17T13:49:56.9122172Z env: 2022-08-17T13:49:56.9122355Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:56.9122534Z ##[endgroup] 2022-08-17T13:49:56.9147358Z ##[group]Run # Remove any previous test jsons if they exist 2022-08-17T13:49:56.9147659Z # Remove any previous test jsons if they exist 2022-08-17T13:49:56.9147899Z rm -f test-jsons-*.zip 2022-08-17T13:49:56.9148141Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-08-17T13:49:56.9159365Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:49:56.9159630Z env: 2022-08-17T13:49:56.9159803Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:56.9160073Z FILE_SUFFIX: test-distributed-1-1-linux.2xlarge_7878166005 2022-08-17T13:49:56.9160321Z ##[endgroup] 2022-08-17T13:49:56.9303509Z adding: test/allowlist_for_publicAPI.json (deflated 80%) 2022-08-17T13:49:56.9339411Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-08-17T13:49:56.9345896Z adding: test/profiler_utils_mock_events.json (deflated 87%) 2022-08-17T13:49:56.9347154Z adding: test/.pytorch-slow-tests.json (deflated 75%) 2022-08-17T13:49:56.9352473Z adding: test/.pytorch-disabled-tests.json (deflated 85%) 2022-08-17T13:49:56.9374010Z ##[group]Run # Remove any previous test reports if they exist 2022-08-17T13:49:56.9374318Z # Remove any previous test reports if they exist 2022-08-17T13:49:56.9374569Z rm -f test-reports-*.zip 2022-08-17T13:49:56.9374818Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' 2022-08-17T13:49:56.9385856Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:49:56.9386086Z env: 2022-08-17T13:49:56.9386255Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:56.9386522Z FILE_SUFFIX: test-distributed-1-1-linux.2xlarge_7878166005 2022-08-17T13:49:56.9386777Z ##[endgroup] 2022-08-17T13:49:56.9578034Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220817121046.xml (deflated 62%) 2022-08-17T13:49:56.9578905Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220817121047.xml (deflated 62%) 2022-08-17T13:49:56.9579673Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220817121048.xml (deflated 74%) 2022-08-17T13:49:56.9580425Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestStorageKeys-20220817121048.xml (deflated 40%) 2022-08-17T13:49:56.9581271Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220817121048.xml (deflated 60%) 2022-08-17T13:49:56.9582108Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220817121053.xml (deflated 42%) 2022-08-17T13:49:56.9583274Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220817121053.xml (deflated 74%) 2022-08-17T13:49:56.9584180Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220817121053.xml (deflated 45%) 2022-08-17T13:49:56.9585157Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220817121055.xml (deflated 68%) 2022-08-17T13:49:56.9586074Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220817121055.xml (deflated 43%) 2022-08-17T13:49:56.9587081Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220817121055.xml (deflated 45%) 2022-08-17T13:49:56.9587906Z adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20220817121105.xml (deflated 72%) 2022-08-17T13:49:56.9588567Z adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220817121107.xml (deflated 61%) 2022-08-17T13:49:56.9589428Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220817121108.xml (deflated 75%) 2022-08-17T13:49:56.9590215Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220817121110.xml (deflated 62%) 2022-08-17T13:49:56.9591094Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220817121111.xml (deflated 76%) 2022-08-17T13:49:56.9591889Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220817121112.xml (deflated 71%) 2022-08-17T13:49:56.9592660Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220817121114.xml (deflated 70%) 2022-08-17T13:49:56.9593531Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220817121116.xml (deflated 88%) 2022-08-17T13:49:56.9594292Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220817121118.xml (deflated 61%) 2022-08-17T13:49:56.9594994Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220817121119.xml (deflated 78%) 2022-08-17T13:49:56.9595698Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220817121120.xml (deflated 45%) 2022-08-17T13:49:56.9596588Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220817121122.xml (deflated 44%) 2022-08-17T13:49:56.9597296Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220817121122.xml (deflated 44%) 2022-08-17T13:49:56.9598045Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220817121122.xml (deflated 62%) 2022-08-17T13:49:56.9598757Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220817121122.xml (deflated 60%) 2022-08-17T13:49:56.9599545Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20220817121122.xml (deflated 60%) 2022-08-17T13:49:56.9600326Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220817121122.xml (deflated 62%) 2022-08-17T13:49:56.9601041Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220817121122.xml (deflated 62%) 2022-08-17T13:49:56.9601774Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220817121122.xml (deflated 90%) 2022-08-17T13:49:56.9602616Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220817121122.xml (deflated 71%) 2022-08-17T13:49:56.9603403Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220817121122.xml (deflated 88%) 2022-08-17T13:49:56.9604222Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220817121122.xml (deflated 86%) 2022-08-17T13:49:56.9605045Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220817121122.xml (deflated 63%) 2022-08-17T13:49:56.9605793Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220817121127.xml (deflated 63%) 2022-08-17T13:49:56.9606799Z adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20220817121128.xml (deflated 78%) 2022-08-17T13:49:56.9607472Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220817121129.xml (deflated 67%) 2022-08-17T13:49:56.9608212Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220817121129.xml (deflated 78%) 2022-08-17T13:49:56.9608841Z adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220817121133.xml (deflated 69%) 2022-08-17T13:49:56.9609575Z adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220817121133.xml (deflated 62%) 2022-08-17T13:49:56.9610435Z adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220817121137.xml (deflated 82%) 2022-08-17T13:49:56.9611146Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220817121207.xml (deflated 80%) 2022-08-17T13:49:56.9611796Z adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220817121223.xml (deflated 63%) 2022-08-17T13:49:56.9612505Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220817121225.xml (deflated 56%) 2022-08-17T13:49:56.9613281Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220817121225.xml (deflated 80%) 2022-08-17T13:49:56.9614104Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220817121225.xml (deflated 79%) 2022-08-17T13:49:56.9614825Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220817121225.xml (deflated 63%) 2022-08-17T13:49:56.9615525Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220817121246.xml (deflated 54%) 2022-08-17T13:49:56.9616269Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220817121257.xml (deflated 71%) 2022-08-17T13:49:56.9617076Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220817121257.xml (deflated 69%) 2022-08-17T13:49:56.9617835Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220817121257.xml (deflated 66%) 2022-08-17T13:49:56.9618599Z adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220817121302.xml (deflated 71%) 2022-08-17T13:49:56.9619308Z adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220817121307.xml (deflated 55%) 2022-08-17T13:49:56.9620035Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220817121308.xml (deflated 62%) 2022-08-17T13:49:56.9620647Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220817121308.xml (deflated 69%) 2022-08-17T13:49:56.9621401Z adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20220817121310.xml (deflated 68%) 2022-08-17T13:49:56.9622092Z adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220817121312.xml (deflated 62%) 2022-08-17T13:49:56.9622922Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220817121316.xml (deflated 81%) 2022-08-17T13:49:56.9623583Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220817121316.xml (deflated 85%) 2022-08-17T13:49:56.9624344Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220817121316.xml (deflated 85%) 2022-08-17T13:49:56.9624978Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220817121318.xml (deflated 64%) 2022-08-17T13:49:56.9625626Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220817121324.xml (deflated 81%) 2022-08-17T13:49:56.9626342Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220817121338.xml (deflated 86%) 2022-08-17T13:49:56.9627074Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220817121338.xml (deflated 86%) 2022-08-17T13:49:56.9627719Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220817121405.xml (deflated 91%) 2022-08-17T13:49:56.9628345Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220817121420.xml (deflated 88%) 2022-08-17T13:49:56.9628940Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220817121437.xml (deflated 81%) 2022-08-17T13:49:56.9629579Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220817121437.xml (deflated 64%) 2022-08-17T13:49:56.9630218Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220817121437.xml (deflated 62%) 2022-08-17T13:49:56.9630813Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220817121437.xml (deflated 93%) 2022-08-17T13:49:56.9631574Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220817121606.xml (deflated 84%) 2022-08-17T13:49:56.9632222Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220817121621.xml (deflated 86%) 2022-08-17T13:49:56.9632849Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220817121636.xml (deflated 45%) 2022-08-17T13:49:56.9633507Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220817121639.xml (deflated 94%) 2022-08-17T13:49:56.9634407Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220817121701.xml (deflated 77%) 2022-08-17T13:49:56.9635143Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220817121710.xml (deflated 59%) 2022-08-17T13:49:56.9635745Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220817121715.xml (deflated 59%) 2022-08-17T13:49:56.9636501Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220817121719.xml (deflated 87%) 2022-08-17T13:49:56.9637932Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220817121732.xml (deflated 80%) 2022-08-17T13:49:56.9639489Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220817121756.xml (deflated 93%) 2022-08-17T13:49:56.9640383Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220817121756.xml (deflated 59%) 2022-08-17T13:49:56.9641139Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220817121911.xml (deflated 42%) 2022-08-17T13:49:56.9641779Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220817121915.xml (deflated 46%) 2022-08-17T13:49:56.9643758Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220817121918.xml (deflated 91%) 2022-08-17T13:49:56.9644559Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220817122016.xml (deflated 43%) 2022-08-17T13:49:56.9645278Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220817122016.xml (deflated 43%) 2022-08-17T13:49:56.9646181Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220817122019.xml (deflated 77%) 2022-08-17T13:49:56.9646876Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220817122028.xml (deflated 56%) 2022-08-17T13:49:56.9647584Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220817122031.xml (deflated 68%) 2022-08-17T13:49:56.9648685Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220817122031.xml (deflated 86%) 2022-08-17T13:49:56.9651482Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220817122046.xml (deflated 95%) 2022-08-17T13:49:56.9653446Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220817122221.xml (deflated 94%) 2022-08-17T13:49:56.9654705Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220817122221.xml (deflated 86%) 2022-08-17T13:49:56.9655708Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220817122411.xml (deflated 43%) 2022-08-17T13:49:56.9656773Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220817122414.xml (deflated 41%) 2022-08-17T13:49:56.9657849Z adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220817122419.xml (deflated 69%) 2022-08-17T13:49:56.9658504Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220817122420.xml (deflated 87%) 2022-08-17T13:49:56.9659058Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220817122420.xml (deflated 91%) 2022-08-17T13:49:56.9659993Z adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220817122500.xml (deflated 64%) 2022-08-17T13:49:56.9660981Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220817122502.xml (deflated 91%) 2022-08-17T13:49:56.9661940Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220817122502.xml (deflated 73%) 2022-08-17T13:49:56.9662922Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220817122623.xml (deflated 41%) 2022-08-17T13:49:56.9663814Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122624.xml (deflated 42%) 2022-08-17T13:49:56.9664647Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122628.xml (deflated 42%) 2022-08-17T13:49:56.9665478Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220817122632.xml (deflated 42%) 2022-08-17T13:49:56.9666323Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122636.xml (deflated 42%) 2022-08-17T13:49:56.9667121Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122639.xml (deflated 42%) 2022-08-17T13:49:56.9667975Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122643.xml (deflated 42%) 2022-08-17T13:49:56.9668794Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220817122647.xml (deflated 42%) 2022-08-17T13:49:56.9669596Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220817122650.xml (deflated 42%) 2022-08-17T13:49:56.9670381Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122654.xml (deflated 40%) 2022-08-17T13:49:56.9671150Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122658.xml (deflated 41%) 2022-08-17T13:49:56.9672014Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122701.xml (deflated 41%) 2022-08-17T13:49:56.9672794Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122705.xml (deflated 41%) 2022-08-17T13:49:56.9673647Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122709.xml (deflated 41%) 2022-08-17T13:49:56.9674396Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122710.xml (deflated 41%) 2022-08-17T13:49:56.9675194Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122712.xml (deflated 41%) 2022-08-17T13:49:56.9675983Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220817122713.xml (deflated 41%) 2022-08-17T13:49:56.9676820Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122715.xml (deflated 44%) 2022-08-17T13:49:56.9677693Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122719.xml (deflated 43%) 2022-08-17T13:49:56.9678689Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122723.xml (deflated 44%) 2022-08-17T13:49:56.9679536Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122726.xml (deflated 44%) 2022-08-17T13:49:56.9680456Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122730.xml (deflated 44%) 2022-08-17T13:49:56.9681418Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122734.xml (deflated 44%) 2022-08-17T13:49:56.9682255Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122738.xml (deflated 44%) 2022-08-17T13:49:56.9683166Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122742.xml (deflated 44%) 2022-08-17T13:49:56.9684039Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122745.xml (deflated 44%) 2022-08-17T13:49:56.9684910Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122749.xml (deflated 45%) 2022-08-17T13:49:56.9685819Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122753.xml (deflated 44%) 2022-08-17T13:49:56.9686854Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122757.xml (deflated 44%) 2022-08-17T13:49:56.9687721Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122801.xml (deflated 44%) 2022-08-17T13:49:56.9688646Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122804.xml (deflated 44%) 2022-08-17T13:49:56.9689559Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122808.xml (deflated 44%) 2022-08-17T13:49:56.9690392Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122812.xml (deflated 44%) 2022-08-17T13:49:56.9691293Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122816.xml (deflated 43%) 2022-08-17T13:49:56.9692206Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122818.xml (deflated 44%) 2022-08-17T13:49:56.9693092Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122822.xml (deflated 44%) 2022-08-17T13:49:56.9693964Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122825.xml (deflated 44%) 2022-08-17T13:49:56.9694867Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122829.xml (deflated 43%) 2022-08-17T13:49:56.9695742Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122833.xml (deflated 44%) 2022-08-17T13:49:56.9696608Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122837.xml (deflated 43%) 2022-08-17T13:49:56.9697605Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122841.xml (deflated 44%) 2022-08-17T13:49:56.9698503Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122845.xml (deflated 43%) 2022-08-17T13:49:56.9699341Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122847.xml (deflated 44%) 2022-08-17T13:49:56.9700319Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122851.xml (deflated 44%) 2022-08-17T13:49:56.9701242Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122855.xml (deflated 44%) 2022-08-17T13:49:56.9702117Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122858.xml (deflated 44%) 2022-08-17T13:49:56.9703035Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122902.xml (deflated 44%) 2022-08-17T13:49:56.9703951Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122906.xml (deflated 44%) 2022-08-17T13:49:56.9704831Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122910.xml (deflated 44%) 2022-08-17T13:49:56.9705699Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122914.xml (deflated 44%) 2022-08-17T13:49:56.9706606Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122918.xml (deflated 44%) 2022-08-17T13:49:56.9707510Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122921.xml (deflated 43%) 2022-08-17T13:49:56.9708361Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122925.xml (deflated 44%) 2022-08-17T13:49:56.9709279Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122929.xml (deflated 43%) 2022-08-17T13:49:56.9710179Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122933.xml (deflated 43%) 2022-08-17T13:49:56.9711007Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122937.xml (deflated 43%) 2022-08-17T13:49:56.9711880Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122940.xml (deflated 44%) 2022-08-17T13:49:56.9712738Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122944.xml (deflated 44%) 2022-08-17T13:49:56.9713648Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122948.xml (deflated 44%) 2022-08-17T13:49:56.9714476Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122952.xml (deflated 44%) 2022-08-17T13:49:56.9715411Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817122956.xml (deflated 43%) 2022-08-17T13:49:56.9716322Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123000.xml (deflated 44%) 2022-08-17T13:49:56.9717217Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123003.xml (deflated 44%) 2022-08-17T13:49:56.9718129Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123007.xml (deflated 44%) 2022-08-17T13:49:56.9719025Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123011.xml (deflated 45%) 2022-08-17T13:49:56.9719916Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123015.xml (deflated 44%) 2022-08-17T13:49:56.9720821Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123019.xml (deflated 44%) 2022-08-17T13:49:56.9721739Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123023.xml (deflated 44%) 2022-08-17T13:49:56.9722615Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123027.xml (deflated 44%) 2022-08-17T13:49:56.9723466Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123030.xml (deflated 44%) 2022-08-17T13:49:56.9724392Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123034.xml (deflated 43%) 2022-08-17T13:49:56.9725299Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123038.xml (deflated 44%) 2022-08-17T13:49:56.9726268Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123042.xml (deflated 44%) 2022-08-17T13:49:56.9727140Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123046.xml (deflated 43%) 2022-08-17T13:49:56.9728067Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123049.xml (deflated 43%) 2022-08-17T13:49:56.9728974Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123053.xml (deflated 44%) 2022-08-17T13:49:56.9729872Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123057.xml (deflated 43%) 2022-08-17T13:49:56.9730709Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123101.xml (deflated 44%) 2022-08-17T13:49:56.9731594Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123105.xml (deflated 44%) 2022-08-17T13:49:56.9732505Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123109.xml (deflated 43%) 2022-08-17T13:49:56.9733402Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123113.xml (deflated 44%) 2022-08-17T13:49:56.9734242Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123117.xml (deflated 43%) 2022-08-17T13:49:56.9735067Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123120.xml (deflated 44%) 2022-08-17T13:49:56.9736023Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123124.xml (deflated 44%) 2022-08-17T13:49:56.9736934Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123128.xml (deflated 44%) 2022-08-17T13:49:56.9737788Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123132.xml (deflated 43%) 2022-08-17T13:49:56.9738715Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123136.xml (deflated 43%) 2022-08-17T13:49:56.9739614Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123140.xml (deflated 43%) 2022-08-17T13:49:56.9740482Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123143.xml (deflated 43%) 2022-08-17T13:49:56.9741346Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123147.xml (deflated 43%) 2022-08-17T13:49:56.9742268Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123151.xml (deflated 43%) 2022-08-17T13:49:56.9743164Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123155.xml (deflated 43%) 2022-08-17T13:49:56.9744088Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123157.xml (deflated 43%) 2022-08-17T13:49:56.9744954Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123200.xml (deflated 43%) 2022-08-17T13:49:56.9745834Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123204.xml (deflated 43%) 2022-08-17T13:49:56.9746742Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123208.xml (deflated 43%) 2022-08-17T13:49:56.9747643Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123212.xml (deflated 43%) 2022-08-17T13:49:56.9748528Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123216.xml (deflated 43%) 2022-08-17T13:49:56.9749389Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123220.xml (deflated 43%) 2022-08-17T13:49:56.9750353Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123224.xml (deflated 43%) 2022-08-17T13:49:56.9751269Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123227.xml (deflated 43%) 2022-08-17T13:49:56.9752102Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123231.xml (deflated 43%) 2022-08-17T13:49:56.9753015Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123235.xml (deflated 44%) 2022-08-17T13:49:56.9753881Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220817123239.xml (deflated 44%) 2022-08-17T13:49:56.9754817Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123243.xml (deflated 44%) 2022-08-17T13:49:56.9755776Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123247.xml (deflated 45%) 2022-08-17T13:49:56.9756714Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220817123250.xml (deflated 44%) 2022-08-17T13:49:56.9757580Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220817123256.xml (deflated 42%) 2022-08-17T13:49:56.9758453Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220817123307.xml (deflated 42%) 2022-08-17T13:49:56.9759211Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123317.xml (deflated 41%) 2022-08-17T13:49:56.9759986Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123327.xml (deflated 41%) 2022-08-17T13:49:56.9760743Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123337.xml (deflated 41%) 2022-08-17T13:49:56.9761485Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123351.xml (deflated 42%) 2022-08-17T13:49:56.9762196Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123401.xml (deflated 41%) 2022-08-17T13:49:56.9762957Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123405.xml (deflated 42%) 2022-08-17T13:49:56.9763662Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123415.xml (deflated 41%) 2022-08-17T13:49:56.9764461Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123424.xml (deflated 41%) 2022-08-17T13:49:56.9765170Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123430.xml (deflated 41%) 2022-08-17T13:49:56.9766082Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123440.xml (deflated 41%) 2022-08-17T13:49:56.9766734Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123450.xml (deflated 41%) 2022-08-17T13:49:56.9767513Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123459.xml (deflated 41%) 2022-08-17T13:49:56.9768222Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123505.xml (deflated 41%) 2022-08-17T13:49:56.9769003Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123522.xml (deflated 41%) 2022-08-17T13:49:56.9769695Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123534.xml (deflated 41%) 2022-08-17T13:49:56.9770466Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123540.xml (deflated 41%) 2022-08-17T13:49:56.9771177Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123550.xml (deflated 41%) 2022-08-17T13:49:56.9771880Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123557.xml (deflated 41%) 2022-08-17T13:49:56.9772580Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123607.xml (deflated 41%) 2022-08-17T13:49:56.9773389Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220817123617.xml (deflated 40%) 2022-08-17T13:49:56.9774115Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123626.xml (deflated 41%) 2022-08-17T13:49:56.9774874Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123636.xml (deflated 41%) 2022-08-17T13:49:56.9775689Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123646.xml (deflated 41%) 2022-08-17T13:49:56.9776452Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123656.xml (deflated 41%) 2022-08-17T13:49:56.9777230Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123702.xml (deflated 41%) 2022-08-17T13:49:56.9777968Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220817123711.xml (deflated 42%) 2022-08-17T13:49:56.9778728Z adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220817123721.xml (deflated 39%) 2022-08-17T13:49:56.9779454Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123726.xml (deflated 42%) 2022-08-17T13:49:56.9780287Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123728.xml (deflated 42%) 2022-08-17T13:49:56.9781104Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123730.xml (deflated 41%) 2022-08-17T13:49:56.9781841Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220817123734.xml (deflated 41%) 2022-08-17T13:49:56.9782774Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123738.xml (deflated 42%) 2022-08-17T13:49:56.9783630Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123744.xml (deflated 42%) 2022-08-17T13:49:56.9784476Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123749.xml (deflated 41%) 2022-08-17T13:49:56.9785367Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220817123755.xml (deflated 41%) 2022-08-17T13:49:56.9786160Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123800.xml (deflated 43%) 2022-08-17T13:49:56.9786947Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123804.xml (deflated 41%) 2022-08-17T13:49:56.9787735Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123809.xml (deflated 41%) 2022-08-17T13:49:56.9788500Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123813.xml (deflated 41%) 2022-08-17T13:49:56.9789306Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123817.xml (deflated 40%) 2022-08-17T13:49:56.9790065Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123822.xml (deflated 41%) 2022-08-17T13:49:56.9790893Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123826.xml (deflated 42%) 2022-08-17T13:49:56.9791755Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123830.xml (deflated 41%) 2022-08-17T13:49:56.9792496Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123834.xml (deflated 42%) 2022-08-17T13:49:56.9793295Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123836.xml (deflated 40%) 2022-08-17T13:49:56.9794088Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123840.xml (deflated 41%) 2022-08-17T13:49:56.9794924Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123842.xml (deflated 41%) 2022-08-17T13:49:56.9795664Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123847.xml (deflated 41%) 2022-08-17T13:49:56.9796507Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123851.xml (deflated 41%) 2022-08-17T13:49:56.9797249Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123855.xml (deflated 40%) 2022-08-17T13:49:56.9798060Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123859.xml (deflated 40%) 2022-08-17T13:49:56.9798849Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123903.xml (deflated 40%) 2022-08-17T13:49:56.9799599Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123907.xml (deflated 40%) 2022-08-17T13:49:56.9800421Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123912.xml (deflated 40%) 2022-08-17T13:49:56.9801166Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123916.xml (deflated 40%) 2022-08-17T13:49:56.9801989Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123920.xml (deflated 41%) 2022-08-17T13:49:56.9802804Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123924.xml (deflated 40%) 2022-08-17T13:49:56.9803565Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123928.xml (deflated 40%) 2022-08-17T13:49:56.9804367Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123942.xml (deflated 42%) 2022-08-17T13:49:56.9805122Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123944.xml (deflated 40%) 2022-08-17T13:49:56.9806008Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123949.xml (deflated 40%) 2022-08-17T13:49:56.9806779Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123953.xml (deflated 40%) 2022-08-17T13:49:56.9807533Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817123957.xml (deflated 40%) 2022-08-17T13:49:56.9808339Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124001.xml (deflated 49%) 2022-08-17T13:49:56.9809106Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124005.xml (deflated 41%) 2022-08-17T13:49:56.9809928Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124010.xml (deflated 42%) 2022-08-17T13:49:56.9810733Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124014.xml (deflated 41%) 2022-08-17T13:49:56.9811478Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124019.xml (deflated 41%) 2022-08-17T13:49:56.9812285Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124023.xml (deflated 41%) 2022-08-17T13:49:56.9813065Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124027.xml (deflated 42%) 2022-08-17T13:49:56.9813882Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124029.xml (deflated 41%) 2022-08-17T13:49:56.9814660Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124033.xml (deflated 41%) 2022-08-17T13:49:56.9815422Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124038.xml (deflated 41%) 2022-08-17T13:49:56.9816234Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124042.xml (deflated 41%) 2022-08-17T13:49:56.9816959Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124047.xml (deflated 40%) 2022-08-17T13:49:56.9817775Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124051.xml (deflated 40%) 2022-08-17T13:49:56.9818523Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124055.xml (deflated 41%) 2022-08-17T13:49:56.9819321Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124100.xml (deflated 41%) 2022-08-17T13:49:56.9820101Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124104.xml (deflated 41%) 2022-08-17T13:49:56.9820845Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124108.xml (deflated 41%) 2022-08-17T13:49:56.9821649Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124113.xml (deflated 40%) 2022-08-17T13:49:56.9822392Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124117.xml (deflated 40%) 2022-08-17T13:49:56.9823207Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124121.xml (deflated 41%) 2022-08-17T13:49:56.9824030Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124126.xml (deflated 41%) 2022-08-17T13:49:56.9824811Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124131.xml (deflated 41%) 2022-08-17T13:49:56.9825580Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124135.xml (deflated 41%) 2022-08-17T13:49:56.9826401Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124139.xml (deflated 41%) 2022-08-17T13:49:56.9827130Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124143.xml (deflated 44%) 2022-08-17T13:49:56.9828006Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124148.xml (deflated 41%) 2022-08-17T13:49:56.9828763Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124152.xml (deflated 41%) 2022-08-17T13:49:56.9829557Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124156.xml (deflated 41%) 2022-08-17T13:49:56.9830322Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124201.xml (deflated 41%) 2022-08-17T13:49:56.9831143Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124205.xml (deflated 41%) 2022-08-17T13:49:56.9831951Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124209.xml (deflated 41%) 2022-08-17T13:49:56.9832703Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124213.xml (deflated 41%) 2022-08-17T13:49:56.9833504Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124219.xml (deflated 41%) 2022-08-17T13:49:56.9834249Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124223.xml (deflated 41%) 2022-08-17T13:49:56.9835074Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220817124227.xml (deflated 41%) 2022-08-17T13:49:56.9835897Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124232.xml (deflated 43%) 2022-08-17T13:49:56.9836641Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124234.xml (deflated 41%) 2022-08-17T13:49:56.9837476Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124238.xml (deflated 42%) 2022-08-17T13:49:56.9838237Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220817124242.xml (deflated 42%) 2022-08-17T13:49:56.9839090Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124244.xml (deflated 41%) 2022-08-17T13:49:56.9839915Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124248.xml (deflated 41%) 2022-08-17T13:49:56.9840678Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124252.xml (deflated 41%) 2022-08-17T13:49:56.9841535Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220817124257.xml (deflated 41%) 2022-08-17T13:49:56.9842303Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124301.xml (deflated 40%) 2022-08-17T13:49:56.9843045Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124305.xml (deflated 39%) 2022-08-17T13:49:56.9843737Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124310.xml (deflated 40%) 2022-08-17T13:49:56.9844489Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124314.xml (deflated 40%) 2022-08-17T13:49:56.9845231Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124318.xml (deflated 40%) 2022-08-17T13:49:56.9846154Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124322.xml (deflated 40%) 2022-08-17T13:49:56.9846888Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124327.xml (deflated 40%) 2022-08-17T13:49:56.9847675Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124331.xml (deflated 40%) 2022-08-17T13:49:56.9848490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124335.xml (deflated 39%) 2022-08-17T13:49:56.9849359Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124339.xml (deflated 40%) 2022-08-17T13:49:56.9850180Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124343.xml (deflated 40%) 2022-08-17T13:49:56.9850953Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124347.xml (deflated 40%) 2022-08-17T13:49:56.9851728Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124351.xml (deflated 39%) 2022-08-17T13:49:56.9852561Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124355.xml (deflated 40%) 2022-08-17T13:49:56.9853333Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124400.xml (deflated 40%) 2022-08-17T13:49:56.9854168Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124404.xml (deflated 40%) 2022-08-17T13:49:56.9854949Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124408.xml (deflated 40%) 2022-08-17T13:49:56.9855771Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124412.xml (deflated 40%) 2022-08-17T13:49:56.9856558Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124417.xml (deflated 40%) 2022-08-17T13:49:56.9857313Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124421.xml (deflated 40%) 2022-08-17T13:49:56.9858125Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124425.xml (deflated 41%) 2022-08-17T13:49:56.9858910Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124429.xml (deflated 40%) 2022-08-17T13:49:56.9859705Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124433.xml (deflated 40%) 2022-08-17T13:49:56.9860505Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124437.xml (deflated 40%) 2022-08-17T13:49:56.9861281Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124442.xml (deflated 40%) 2022-08-17T13:49:56.9862108Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124446.xml (deflated 40%) 2022-08-17T13:49:56.9862967Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124450.xml (deflated 40%) 2022-08-17T13:49:56.9863773Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124454.xml (deflated 39%) 2022-08-17T13:49:56.9864578Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124459.xml (deflated 40%) 2022-08-17T13:49:56.9865495Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124503.xml (deflated 40%) 2022-08-17T13:49:56.9866271Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124507.xml (deflated 40%) 2022-08-17T13:49:56.9867079Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124511.xml (deflated 39%) 2022-08-17T13:49:56.9867861Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124515.xml (deflated 39%) 2022-08-17T13:49:56.9868738Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124520.xml (deflated 40%) 2022-08-17T13:49:56.9869514Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124525.xml (deflated 40%) 2022-08-17T13:49:56.9870335Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124529.xml (deflated 40%) 2022-08-17T13:49:56.9871113Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124533.xml (deflated 40%) 2022-08-17T13:49:56.9871937Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124537.xml (deflated 41%) 2022-08-17T13:49:56.9872719Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124541.xml (deflated 40%) 2022-08-17T13:49:56.9873490Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124546.xml (deflated 39%) 2022-08-17T13:49:56.9874304Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124550.xml (deflated 40%) 2022-08-17T13:49:56.9875074Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124554.xml (deflated 40%) 2022-08-17T13:49:56.9875905Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124559.xml (deflated 40%) 2022-08-17T13:49:56.9876674Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124603.xml (deflated 40%) 2022-08-17T13:49:56.9877485Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124607.xml (deflated 40%) 2022-08-17T13:49:56.9878266Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124611.xml (deflated 40%) 2022-08-17T13:49:56.9879087Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124615.xml (deflated 40%) 2022-08-17T13:49:56.9879874Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124620.xml (deflated 40%) 2022-08-17T13:49:56.9880622Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124624.xml (deflated 40%) 2022-08-17T13:49:56.9881464Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124628.xml (deflated 40%) 2022-08-17T13:49:56.9882235Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124632.xml (deflated 40%) 2022-08-17T13:49:56.9883157Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124636.xml (deflated 40%) 2022-08-17T13:49:56.9883974Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124641.xml (deflated 40%) 2022-08-17T13:49:56.9884804Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220817124645.xml (deflated 40%) 2022-08-17T13:49:56.9885684Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220817124649.xml (deflated 43%) 2022-08-17T13:49:56.9886714Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220817124654.xml (deflated 43%) 2022-08-17T13:49:56.9887593Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124658.xml (deflated 41%) 2022-08-17T13:49:56.9888553Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124702.xml (deflated 41%) 2022-08-17T13:49:56.9889432Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124706.xml (deflated 41%) 2022-08-17T13:49:56.9890316Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124710.xml (deflated 41%) 2022-08-17T13:49:56.9891148Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124714.xml (deflated 40%) 2022-08-17T13:49:56.9892042Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124718.xml (deflated 41%) 2022-08-17T13:49:56.9892902Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124722.xml (deflated 42%) 2022-08-17T13:49:56.9893731Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124726.xml (deflated 42%) 2022-08-17T13:49:56.9894602Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124730.xml (deflated 41%) 2022-08-17T13:49:56.9895426Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124733.xml (deflated 41%) 2022-08-17T13:49:56.9896287Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124737.xml (deflated 41%) 2022-08-17T13:49:56.9897095Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124741.xml (deflated 41%) 2022-08-17T13:49:56.9897947Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220817124745.xml (deflated 41%) 2022-08-17T13:49:56.9898747Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124749.xml (deflated 40%) 2022-08-17T13:49:56.9899517Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124753.xml (deflated 39%) 2022-08-17T13:49:56.9900485Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124757.xml (deflated 40%) 2022-08-17T13:49:56.9901345Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124801.xml (deflated 40%) 2022-08-17T13:49:56.9902218Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124806.xml (deflated 40%) 2022-08-17T13:49:56.9903159Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124819.xml (deflated 40%) 2022-08-17T13:49:56.9904045Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124823.xml (deflated 40%) 2022-08-17T13:49:56.9904936Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124827.xml (deflated 40%) 2022-08-17T13:49:56.9905929Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124831.xml (deflated 40%) 2022-08-17T13:49:56.9906707Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124836.xml (deflated 40%) 2022-08-17T13:49:56.9907622Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124840.xml (deflated 40%) 2022-08-17T13:49:56.9908505Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124844.xml (deflated 40%) 2022-08-17T13:49:56.9909413Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124848.xml (deflated 40%) 2022-08-17T13:49:56.9910194Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124852.xml (deflated 39%) 2022-08-17T13:49:56.9911085Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124856.xml (deflated 40%) 2022-08-17T13:49:56.9911934Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124900.xml (deflated 40%) 2022-08-17T13:49:56.9912787Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124905.xml (deflated 39%) 2022-08-17T13:49:56.9913564Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124909.xml (deflated 40%) 2022-08-17T13:49:56.9914570Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124913.xml (deflated 39%) 2022-08-17T13:49:56.9915458Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124917.xml (deflated 40%) 2022-08-17T13:49:56.9916291Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124921.xml (deflated 40%) 2022-08-17T13:49:56.9917199Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124925.xml (deflated 40%) 2022-08-17T13:49:56.9918012Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124930.xml (deflated 41%) 2022-08-17T13:49:56.9918867Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124931.xml (deflated 40%) 2022-08-17T13:49:56.9919732Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124936.xml (deflated 39%) 2022-08-17T13:49:56.9920660Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124940.xml (deflated 40%) 2022-08-17T13:49:56.9921485Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124944.xml (deflated 40%) 2022-08-17T13:49:56.9922327Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124948.xml (deflated 39%) 2022-08-17T13:49:56.9941816Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124952.xml (deflated 40%) 2022-08-17T13:49:56.9943243Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817124956.xml (deflated 39%) 2022-08-17T13:49:56.9943965Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125000.xml (deflated 39%) 2022-08-17T13:49:56.9944594Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125004.xml (deflated 40%) 2022-08-17T13:49:56.9945355Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125010.xml (deflated 40%) 2022-08-17T13:49:56.9945986Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125015.xml (deflated 40%) 2022-08-17T13:49:56.9946595Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125019.xml (deflated 40%) 2022-08-17T13:49:56.9947212Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125023.xml (deflated 40%) 2022-08-17T13:49:56.9947865Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125027.xml (deflated 40%) 2022-08-17T13:49:56.9948480Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125031.xml (deflated 40%) 2022-08-17T13:49:56.9949084Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125035.xml (deflated 42%) 2022-08-17T13:49:56.9949692Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125039.xml (deflated 40%) 2022-08-17T13:49:56.9950293Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125044.xml (deflated 40%) 2022-08-17T13:49:56.9950902Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125048.xml (deflated 40%) 2022-08-17T13:49:56.9951495Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125052.xml (deflated 40%) 2022-08-17T13:49:56.9952117Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125056.xml (deflated 40%) 2022-08-17T13:49:56.9952732Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125100.xml (deflated 40%) 2022-08-17T13:49:56.9953341Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125104.xml (deflated 40%) 2022-08-17T13:49:56.9953934Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125108.xml (deflated 40%) 2022-08-17T13:49:56.9954536Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125113.xml (deflated 39%) 2022-08-17T13:49:56.9955149Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125117.xml (deflated 40%) 2022-08-17T13:49:56.9955750Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125121.xml (deflated 40%) 2022-08-17T13:49:56.9956343Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125126.xml (deflated 40%) 2022-08-17T13:49:56.9956950Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125130.xml (deflated 44%) 2022-08-17T13:49:56.9957561Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125136.xml (deflated 40%) 2022-08-17T13:49:56.9958163Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125140.xml (deflated 40%) 2022-08-17T13:49:56.9958755Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125144.xml (deflated 40%) 2022-08-17T13:49:56.9959370Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125148.xml (deflated 40%) 2022-08-17T13:49:56.9959981Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125152.xml (deflated 40%) 2022-08-17T13:49:56.9960627Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125156.xml (deflated 40%) 2022-08-17T13:49:56.9961230Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125200.xml (deflated 40%) 2022-08-17T13:49:56.9961823Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125204.xml (deflated 40%) 2022-08-17T13:49:56.9962462Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125209.xml (deflated 40%) 2022-08-17T13:49:56.9963071Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125213.xml (deflated 40%) 2022-08-17T13:49:56.9963677Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125217.xml (deflated 41%) 2022-08-17T13:49:56.9964269Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125219.xml (deflated 40%) 2022-08-17T13:49:56.9964879Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125223.xml (deflated 41%) 2022-08-17T13:49:56.9965484Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125227.xml (deflated 40%) 2022-08-17T13:49:56.9966214Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125232.xml (deflated 40%) 2022-08-17T13:49:56.9966815Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125236.xml (deflated 40%) 2022-08-17T13:49:56.9967424Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125240.xml (deflated 40%) 2022-08-17T13:49:56.9968032Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125244.xml (deflated 40%) 2022-08-17T13:49:56.9968638Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125249.xml (deflated 40%) 2022-08-17T13:49:56.9969228Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125253.xml (deflated 40%) 2022-08-17T13:49:56.9969843Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125257.xml (deflated 40%) 2022-08-17T13:49:56.9970450Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125301.xml (deflated 40%) 2022-08-17T13:49:56.9971053Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125305.xml (deflated 39%) 2022-08-17T13:49:56.9971649Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125309.xml (deflated 39%) 2022-08-17T13:49:56.9972247Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125313.xml (deflated 39%) 2022-08-17T13:49:56.9972853Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125318.xml (deflated 40%) 2022-08-17T13:49:56.9973464Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125322.xml (deflated 40%) 2022-08-17T13:49:56.9974064Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125326.xml (deflated 40%) 2022-08-17T13:49:56.9974671Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125330.xml (deflated 40%) 2022-08-17T13:49:56.9975342Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125334.xml (deflated 40%) 2022-08-17T13:49:56.9975953Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125339.xml (deflated 40%) 2022-08-17T13:49:56.9976547Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125343.xml (deflated 40%) 2022-08-17T13:49:56.9977153Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125347.xml (deflated 40%) 2022-08-17T13:49:56.9977799Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125351.xml (deflated 40%) 2022-08-17T13:49:56.9978409Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125356.xml (deflated 41%) 2022-08-17T13:49:56.9979002Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125400.xml (deflated 40%) 2022-08-17T13:49:56.9979604Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125404.xml (deflated 40%) 2022-08-17T13:49:56.9980215Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125408.xml (deflated 40%) 2022-08-17T13:49:56.9980826Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125412.xml (deflated 40%) 2022-08-17T13:49:56.9981435Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125416.xml (deflated 40%) 2022-08-17T13:49:56.9982032Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125421.xml (deflated 41%) 2022-08-17T13:49:56.9982735Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125425.xml (deflated 40%) 2022-08-17T13:49:56.9983352Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125429.xml (deflated 40%) 2022-08-17T13:49:56.9983958Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125433.xml (deflated 40%) 2022-08-17T13:49:56.9984555Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125437.xml (deflated 40%) 2022-08-17T13:49:56.9985169Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125441.xml (deflated 40%) 2022-08-17T13:49:56.9985781Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125445.xml (deflated 40%) 2022-08-17T13:49:56.9986391Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125450.xml (deflated 40%) 2022-08-17T13:49:56.9986986Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125454.xml (deflated 39%) 2022-08-17T13:49:56.9987615Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125458.xml (deflated 40%) 2022-08-17T13:49:56.9988228Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125502.xml (deflated 40%) 2022-08-17T13:49:56.9988842Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125509.xml (deflated 39%) 2022-08-17T13:49:56.9989440Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125516.xml (deflated 40%) 2022-08-17T13:49:56.9990094Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125521.xml (deflated 40%) 2022-08-17T13:49:56.9990706Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125525.xml (deflated 40%) 2022-08-17T13:49:56.9991319Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125529.xml (deflated 39%) 2022-08-17T13:49:56.9991913Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125533.xml (deflated 40%) 2022-08-17T13:49:56.9992540Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125540.xml (deflated 40%) 2022-08-17T13:49:56.9993152Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125548.xml (deflated 40%) 2022-08-17T13:49:56.9993769Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125552.xml (deflated 40%) 2022-08-17T13:49:56.9994368Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125556.xml (deflated 40%) 2022-08-17T13:49:56.9994978Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125600.xml (deflated 40%) 2022-08-17T13:49:56.9995585Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125604.xml (deflated 40%) 2022-08-17T13:49:56.9996190Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125609.xml (deflated 40%) 2022-08-17T13:49:56.9996783Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125613.xml (deflated 40%) 2022-08-17T13:49:56.9997385Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125617.xml (deflated 39%) 2022-08-17T13:49:56.9998001Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125621.xml (deflated 39%) 2022-08-17T13:49:56.9998609Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125627.xml (deflated 40%) 2022-08-17T13:49:56.9999220Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125633.xml (deflated 40%) 2022-08-17T13:49:56.9999810Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125637.xml (deflated 40%) 2022-08-17T13:49:57.0000422Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125641.xml (deflated 40%) 2022-08-17T13:49:57.0001033Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125646.xml (deflated 40%) 2022-08-17T13:49:57.0001642Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125650.xml (deflated 41%) 2022-08-17T13:49:57.0002241Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125654.xml (deflated 40%) 2022-08-17T13:49:57.0002851Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125658.xml (deflated 40%) 2022-08-17T13:49:57.0003463Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125702.xml (deflated 40%) 2022-08-17T13:49:57.0004120Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125706.xml (deflated 40%) 2022-08-17T13:49:57.0004723Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125710.xml (deflated 40%) 2022-08-17T13:49:57.0005372Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125715.xml (deflated 40%) 2022-08-17T13:49:57.0006069Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125719.xml (deflated 40%) 2022-08-17T13:49:57.0006684Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125723.xml (deflated 40%) 2022-08-17T13:49:57.0007329Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125727.xml (deflated 40%) 2022-08-17T13:49:57.0007941Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125731.xml (deflated 41%) 2022-08-17T13:49:57.0008550Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125736.xml (deflated 41%) 2022-08-17T13:49:57.0009158Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125740.xml (deflated 41%) 2022-08-17T13:49:57.0009748Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125744.xml (deflated 39%) 2022-08-17T13:49:57.0010348Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125748.xml (deflated 40%) 2022-08-17T13:49:57.0010958Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125752.xml (deflated 41%) 2022-08-17T13:49:57.0011566Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125756.xml (deflated 40%) 2022-08-17T13:49:57.0012163Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125801.xml (deflated 40%) 2022-08-17T13:49:57.0012774Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125805.xml (deflated 40%) 2022-08-17T13:49:57.0013386Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125809.xml (deflated 40%) 2022-08-17T13:49:57.0013993Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125813.xml (deflated 40%) 2022-08-17T13:49:57.0014584Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125817.xml (deflated 40%) 2022-08-17T13:49:57.0015188Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125821.xml (deflated 40%) 2022-08-17T13:49:57.0015799Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125825.xml (deflated 40%) 2022-08-17T13:49:57.0016417Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125830.xml (deflated 40%) 2022-08-17T13:49:57.0017023Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125834.xml (deflated 40%) 2022-08-17T13:49:57.0017620Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125838.xml (deflated 40%) 2022-08-17T13:49:57.0018232Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125842.xml (deflated 40%) 2022-08-17T13:49:57.0018848Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125846.xml (deflated 40%) 2022-08-17T13:49:57.0019454Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125852.xml (deflated 40%) 2022-08-17T13:49:57.0020103Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125857.xml (deflated 40%) 2022-08-17T13:49:57.0020716Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125901.xml (deflated 40%) 2022-08-17T13:49:57.0021330Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125905.xml (deflated 40%) 2022-08-17T13:49:57.0021936Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125915.xml (deflated 39%) 2022-08-17T13:49:57.0022556Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125920.xml (deflated 40%) 2022-08-17T13:49:57.0023261Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125924.xml (deflated 40%) 2022-08-17T13:49:57.0023876Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125928.xml (deflated 40%) 2022-08-17T13:49:57.0024482Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125932.xml (deflated 40%) 2022-08-17T13:49:57.0025074Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125937.xml (deflated 40%) 2022-08-17T13:49:57.0025695Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125941.xml (deflated 40%) 2022-08-17T13:49:57.0026307Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125945.xml (deflated 40%) 2022-08-17T13:49:57.0026920Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125949.xml (deflated 40%) 2022-08-17T13:49:57.0027523Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125953.xml (deflated 40%) 2022-08-17T13:49:57.0028134Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817125957.xml (deflated 39%) 2022-08-17T13:49:57.0028739Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130002.xml (deflated 40%) 2022-08-17T13:49:57.0029344Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130006.xml (deflated 40%) 2022-08-17T13:49:57.0029936Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130012.xml (deflated 40%) 2022-08-17T13:49:57.0030552Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130016.xml (deflated 40%) 2022-08-17T13:49:57.0031166Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130020.xml (deflated 40%) 2022-08-17T13:49:57.0031778Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130025.xml (deflated 39%) 2022-08-17T13:49:57.0032371Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130029.xml (deflated 40%) 2022-08-17T13:49:57.0032986Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130034.xml (deflated 40%) 2022-08-17T13:49:57.0033598Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130038.xml (deflated 40%) 2022-08-17T13:49:57.0034203Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130043.xml (deflated 40%) 2022-08-17T13:49:57.0034799Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130047.xml (deflated 40%) 2022-08-17T13:49:57.0035691Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130051.xml (deflated 40%) 2022-08-17T13:49:57.0036309Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130055.xml (deflated 41%) 2022-08-17T13:49:57.0036913Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130059.xml (deflated 40%) 2022-08-17T13:49:57.0037560Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130104.xml (deflated 41%) 2022-08-17T13:49:57.0038155Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130108.xml (deflated 41%) 2022-08-17T13:49:57.0038764Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130112.xml (deflated 40%) 2022-08-17T13:49:57.0039373Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130116.xml (deflated 40%) 2022-08-17T13:49:57.0039982Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130120.xml (deflated 40%) 2022-08-17T13:49:57.0040578Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130124.xml (deflated 40%) 2022-08-17T13:49:57.0041196Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130129.xml (deflated 39%) 2022-08-17T13:49:57.0041831Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130133.xml (deflated 40%) 2022-08-17T13:49:57.0042624Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130137.xml (deflated 40%) 2022-08-17T13:49:57.0043221Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130141.xml (deflated 40%) 2022-08-17T13:49:57.0043829Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130146.xml (deflated 40%) 2022-08-17T13:49:57.0044448Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130150.xml (deflated 40%) 2022-08-17T13:49:57.0045056Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130154.xml (deflated 39%) 2022-08-17T13:49:57.0045656Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130158.xml (deflated 40%) 2022-08-17T13:49:57.0046379Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130202.xml (deflated 40%) 2022-08-17T13:49:57.0047000Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130207.xml (deflated 40%) 2022-08-17T13:49:57.0047615Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130211.xml (deflated 40%) 2022-08-17T13:49:57.0048210Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130215.xml (deflated 40%) 2022-08-17T13:49:57.0048814Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130219.xml (deflated 40%) 2022-08-17T13:49:57.0049426Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130224.xml (deflated 40%) 2022-08-17T13:49:57.0050031Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130228.xml (deflated 40%) 2022-08-17T13:49:57.0050710Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130232.xml (deflated 40%) 2022-08-17T13:49:57.0051317Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130236.xml (deflated 40%) 2022-08-17T13:49:57.0051933Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130241.xml (deflated 40%) 2022-08-17T13:49:57.0052542Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130245.xml (deflated 39%) 2022-08-17T13:49:57.0053191Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130249.xml (deflated 40%) 2022-08-17T13:49:57.0053789Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130254.xml (deflated 39%) 2022-08-17T13:49:57.0054401Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130301.xml (deflated 40%) 2022-08-17T13:49:57.0055009Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130305.xml (deflated 40%) 2022-08-17T13:49:57.0055612Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130309.xml (deflated 40%) 2022-08-17T13:49:57.0056203Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130314.xml (deflated 40%) 2022-08-17T13:49:57.0056816Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220817130318.xml (deflated 40%) 2022-08-17T13:49:57.0057520Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130321.xml (deflated 44%) 2022-08-17T13:49:57.0058313Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130325.xml (deflated 43%) 2022-08-17T13:49:57.0059090Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130330.xml (deflated 44%) 2022-08-17T13:49:57.0059874Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130334.xml (deflated 44%) 2022-08-17T13:49:57.0060671Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130338.xml (deflated 44%) 2022-08-17T13:49:57.0061451Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130343.xml (deflated 44%) 2022-08-17T13:49:57.0062223Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130347.xml (deflated 43%) 2022-08-17T13:49:57.0063073Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130352.xml (deflated 43%) 2022-08-17T13:49:57.0063862Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130356.xml (deflated 43%) 2022-08-17T13:49:57.0064637Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130400.xml (deflated 44%) 2022-08-17T13:49:57.0065423Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130405.xml (deflated 43%) 2022-08-17T13:49:57.0066187Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130409.xml (deflated 44%) 2022-08-17T13:49:57.0067040Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130413.xml (deflated 44%) 2022-08-17T13:49:57.0067822Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130418.xml (deflated 44%) 2022-08-17T13:49:57.0068594Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130422.xml (deflated 44%) 2022-08-17T13:49:57.0069395Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130426.xml (deflated 43%) 2022-08-17T13:49:57.0070187Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130431.xml (deflated 44%) 2022-08-17T13:49:57.0070971Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130435.xml (deflated 43%) 2022-08-17T13:49:57.0071753Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130440.xml (deflated 44%) 2022-08-17T13:49:57.0072537Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130444.xml (deflated 43%) 2022-08-17T13:49:57.0073307Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130449.xml (deflated 43%) 2022-08-17T13:49:57.0074086Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130453.xml (deflated 43%) 2022-08-17T13:49:57.0074863Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130458.xml (deflated 43%) 2022-08-17T13:49:57.0075625Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130502.xml (deflated 44%) 2022-08-17T13:49:57.0076383Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130507.xml (deflated 44%) 2022-08-17T13:49:57.0077161Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130512.xml (deflated 43%) 2022-08-17T13:49:57.0077940Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130516.xml (deflated 44%) 2022-08-17T13:49:57.0078716Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130521.xml (deflated 44%) 2022-08-17T13:49:57.0079509Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130525.xml (deflated 44%) 2022-08-17T13:49:57.0080276Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130529.xml (deflated 44%) 2022-08-17T13:49:57.0081057Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220817130534.xml (deflated 44%) 2022-08-17T13:49:57.0081815Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130539.xml (deflated 43%) 2022-08-17T13:49:57.0082544Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130543.xml (deflated 43%) 2022-08-17T13:49:57.0083292Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130547.xml (deflated 43%) 2022-08-17T13:49:57.0084010Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130551.xml (deflated 42%) 2022-08-17T13:49:57.0084731Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130556.xml (deflated 43%) 2022-08-17T13:49:57.0085475Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130600.xml (deflated 44%) 2022-08-17T13:49:57.0086274Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130604.xml (deflated 43%) 2022-08-17T13:49:57.0086992Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130608.xml (deflated 42%) 2022-08-17T13:49:57.0087712Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130612.xml (deflated 43%) 2022-08-17T13:49:57.0088428Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130616.xml (deflated 43%) 2022-08-17T13:49:57.0089149Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130620.xml (deflated 43%) 2022-08-17T13:49:57.0089861Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130624.xml (deflated 43%) 2022-08-17T13:49:57.0090572Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130628.xml (deflated 43%) 2022-08-17T13:49:57.0091291Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130634.xml (deflated 43%) 2022-08-17T13:49:57.0092010Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130638.xml (deflated 43%) 2022-08-17T13:49:57.0092704Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130643.xml (deflated 43%) 2022-08-17T13:49:57.0093420Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130647.xml (deflated 43%) 2022-08-17T13:49:57.0094135Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130651.xml (deflated 43%) 2022-08-17T13:49:57.0094845Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130656.xml (deflated 42%) 2022-08-17T13:49:57.0095551Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130700.xml (deflated 43%) 2022-08-17T13:49:57.0096266Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130704.xml (deflated 43%) 2022-08-17T13:49:57.0096978Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130709.xml (deflated 43%) 2022-08-17T13:49:57.0097693Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130713.xml (deflated 43%) 2022-08-17T13:49:57.0098409Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130717.xml (deflated 43%) 2022-08-17T13:49:57.0099162Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130724.xml (deflated 43%) 2022-08-17T13:49:57.0099874Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130730.xml (deflated 43%) 2022-08-17T13:49:57.0100586Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130747.xml (deflated 43%) 2022-08-17T13:49:57.0101307Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130751.xml (deflated 44%) 2022-08-17T13:49:57.0102193Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130755.xml (deflated 43%) 2022-08-17T13:49:57.0103190Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130800.xml (deflated 44%) 2022-08-17T13:49:57.0104128Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130804.xml (deflated 43%) 2022-08-17T13:49:57.0105010Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130808.xml (deflated 43%) 2022-08-17T13:49:57.0105914Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130812.xml (deflated 42%) 2022-08-17T13:49:57.0106835Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130817.xml (deflated 43%) 2022-08-17T13:49:57.0107715Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130821.xml (deflated 44%) 2022-08-17T13:49:57.0108665Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130825.xml (deflated 44%) 2022-08-17T13:49:57.0109541Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130829.xml (deflated 43%) 2022-08-17T13:49:57.0110474Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130833.xml (deflated 43%) 2022-08-17T13:49:57.0111335Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220817130838.xml (deflated 43%) 2022-08-17T13:49:57.0112266Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130842.xml (deflated 43%) 2022-08-17T13:49:57.0113204Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130846.xml (deflated 42%) 2022-08-17T13:49:57.0114152Z adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220817130850.xml (deflated 43%) 2022-08-17T13:49:57.0114950Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220817130855.xml (deflated 38%) 2022-08-17T13:49:57.0115694Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130858.xml (deflated 42%) 2022-08-17T13:49:57.0116501Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130900.xml (deflated 41%) 2022-08-17T13:49:57.0117332Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130901.xml (deflated 41%) 2022-08-17T13:49:57.0118154Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220817130903.xml (deflated 42%) 2022-08-17T13:49:57.0119118Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130905.xml (deflated 41%) 2022-08-17T13:49:57.0119951Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130908.xml (deflated 42%) 2022-08-17T13:49:57.0120831Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130912.xml (deflated 41%) 2022-08-17T13:49:57.0121661Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220817130915.xml (deflated 41%) 2022-08-17T13:49:57.0122495Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130920.xml (deflated 39%) 2022-08-17T13:49:57.0123164Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130923.xml (deflated 40%) 2022-08-17T13:49:57.0123787Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130926.xml (deflated 38%) 2022-08-17T13:49:57.0124441Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130930.xml (deflated 40%) 2022-08-17T13:49:57.0125106Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130933.xml (deflated 39%) 2022-08-17T13:49:57.0125770Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130936.xml (deflated 40%) 2022-08-17T13:49:57.0126555Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130939.xml (deflated 39%) 2022-08-17T13:49:57.0127236Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220817130942.xml (deflated 39%) 2022-08-17T13:49:57.0127974Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130945.xml (deflated 44%) 2022-08-17T13:49:57.0128787Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130949.xml (deflated 44%) 2022-08-17T13:49:57.0129584Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130952.xml (deflated 43%) 2022-08-17T13:49:57.0130369Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130955.xml (deflated 42%) 2022-08-17T13:49:57.0131143Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817130958.xml (deflated 45%) 2022-08-17T13:49:57.0131931Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131001.xml (deflated 45%) 2022-08-17T13:49:57.0132717Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131004.xml (deflated 46%) 2022-08-17T13:49:57.0133509Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131008.xml (deflated 46%) 2022-08-17T13:49:57.0134308Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131011.xml (deflated 45%) 2022-08-17T13:49:57.0135070Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131014.xml (deflated 45%) 2022-08-17T13:49:57.0135848Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131017.xml (deflated 45%) 2022-08-17T13:49:57.0136619Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131020.xml (deflated 44%) 2022-08-17T13:49:57.0137400Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131023.xml (deflated 44%) 2022-08-17T13:49:57.0138179Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131026.xml (deflated 44%) 2022-08-17T13:49:57.0139043Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131030.xml (deflated 44%) 2022-08-17T13:49:57.0139794Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131033.xml (deflated 45%) 2022-08-17T13:49:57.0140620Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131036.xml (deflated 44%) 2022-08-17T13:49:57.0141470Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131039.xml (deflated 45%) 2022-08-17T13:49:57.0142255Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131042.xml (deflated 45%) 2022-08-17T13:49:57.0143108Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131046.xml (deflated 50%) 2022-08-17T13:49:57.0143968Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131049.xml (deflated 42%) 2022-08-17T13:49:57.0144738Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131052.xml (deflated 42%) 2022-08-17T13:49:57.0145521Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131055.xml (deflated 42%) 2022-08-17T13:49:57.0146355Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131058.xml (deflated 42%) 2022-08-17T13:49:57.0147147Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131101.xml (deflated 43%) 2022-08-17T13:49:57.0147932Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131105.xml (deflated 42%) 2022-08-17T13:49:57.0148776Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131108.xml (deflated 42%) 2022-08-17T13:49:57.0149547Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131111.xml (deflated 42%) 2022-08-17T13:49:57.0150377Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131114.xml (deflated 41%) 2022-08-17T13:49:57.0151183Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131117.xml (deflated 45%) 2022-08-17T13:49:57.0151982Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131121.xml (deflated 46%) 2022-08-17T13:49:57.0152800Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131124.xml (deflated 41%) 2022-08-17T13:49:57.0153583Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131127.xml (deflated 42%) 2022-08-17T13:49:57.0154414Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131130.xml (deflated 42%) 2022-08-17T13:49:57.0155214Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131133.xml (deflated 42%) 2022-08-17T13:49:57.0155995Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131137.xml (deflated 42%) 2022-08-17T13:49:57.0156766Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220817131140.xml (deflated 42%) 2022-08-17T13:49:57.0157575Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131143.xml (deflated 40%) 2022-08-17T13:49:57.0158388Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131146.xml (deflated 41%) 2022-08-17T13:49:57.0159129Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131150.xml (deflated 40%) 2022-08-17T13:49:57.0159854Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131153.xml (deflated 40%) 2022-08-17T13:49:57.0160586Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131156.xml (deflated 40%) 2022-08-17T13:49:57.0161358Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131200.xml (deflated 41%) 2022-08-17T13:49:57.0162108Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131203.xml (deflated 40%) 2022-08-17T13:49:57.0162843Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131208.xml (deflated 42%) 2022-08-17T13:49:57.0163592Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131211.xml (deflated 40%) 2022-08-17T13:49:57.0164335Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131214.xml (deflated 41%) 2022-08-17T13:49:57.0165079Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131218.xml (deflated 40%) 2022-08-17T13:49:57.0165796Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131221.xml (deflated 40%) 2022-08-17T13:49:57.0166705Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131225.xml (deflated 40%) 2022-08-17T13:49:57.0167442Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131228.xml (deflated 40%) 2022-08-17T13:49:57.0168163Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131231.xml (deflated 40%) 2022-08-17T13:49:57.0168884Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131235.xml (deflated 40%) 2022-08-17T13:49:57.0169624Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131238.xml (deflated 41%) 2022-08-17T13:49:57.0170357Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131242.xml (deflated 40%) 2022-08-17T13:49:57.0171098Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131246.xml (deflated 40%) 2022-08-17T13:49:57.0171824Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131250.xml (deflated 41%) 2022-08-17T13:49:57.0172555Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131253.xml (deflated 40%) 2022-08-17T13:49:57.0173294Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131257.xml (deflated 40%) 2022-08-17T13:49:57.0174031Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131300.xml (deflated 42%) 2022-08-17T13:49:57.0174753Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131304.xml (deflated 40%) 2022-08-17T13:49:57.0175491Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131307.xml (deflated 41%) 2022-08-17T13:49:57.0176223Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131311.xml (deflated 42%) 2022-08-17T13:49:57.0176944Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131314.xml (deflated 40%) 2022-08-17T13:49:57.0177739Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131318.xml (deflated 40%) 2022-08-17T13:49:57.0178481Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131321.xml (deflated 41%) 2022-08-17T13:49:57.0179222Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131324.xml (deflated 40%) 2022-08-17T13:49:57.0179964Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131328.xml (deflated 40%) 2022-08-17T13:49:57.0180728Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131331.xml (deflated 40%) 2022-08-17T13:49:57.0181468Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131336.xml (deflated 41%) 2022-08-17T13:49:57.0182206Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131340.xml (deflated 40%) 2022-08-17T13:49:57.0183056Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131343.xml (deflated 40%) 2022-08-17T13:49:57.0183786Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131346.xml (deflated 41%) 2022-08-17T13:49:57.0184504Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131350.xml (deflated 40%) 2022-08-17T13:49:57.0185228Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131353.xml (deflated 40%) 2022-08-17T13:49:57.0185978Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131358.xml (deflated 41%) 2022-08-17T13:49:57.0186705Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131401.xml (deflated 40%) 2022-08-17T13:49:57.0187429Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131405.xml (deflated 40%) 2022-08-17T13:49:57.0188167Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131409.xml (deflated 40%) 2022-08-17T13:49:57.0188918Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131412.xml (deflated 41%) 2022-08-17T13:49:57.0189641Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131415.xml (deflated 40%) 2022-08-17T13:49:57.0190362Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131419.xml (deflated 40%) 2022-08-17T13:49:57.0191087Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131424.xml (deflated 42%) 2022-08-17T13:49:57.0191846Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131425.xml (deflated 40%) 2022-08-17T13:49:57.0192558Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131429.xml (deflated 42%) 2022-08-17T13:49:57.0193295Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131430.xml (deflated 41%) 2022-08-17T13:49:57.0194019Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220817131433.xml (deflated 40%) 2022-08-17T13:49:57.0194736Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131437.xml (deflated 39%) 2022-08-17T13:49:57.0195368Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131438.xml (deflated 39%) 2022-08-17T13:49:57.0196073Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131440.xml (deflated 39%) 2022-08-17T13:49:57.0196772Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131441.xml (deflated 40%) 2022-08-17T13:49:57.0197456Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131442.xml (deflated 38%) 2022-08-17T13:49:57.0198121Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220817131443.xml (deflated 39%) 2022-08-17T13:49:57.0198854Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220817131445.xml (deflated 39%) 2022-08-17T13:49:57.0199559Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220817131447.xml (deflated 41%) 2022-08-17T13:49:57.0200223Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131450.xml (deflated 40%) 2022-08-17T13:49:57.0200824Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131451.xml (deflated 40%) 2022-08-17T13:49:57.0201501Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131452.xml (deflated 40%) 2022-08-17T13:49:57.0202173Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131454.xml (deflated 40%) 2022-08-17T13:49:57.0202849Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131455.xml (deflated 39%) 2022-08-17T13:49:57.0203504Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131456.xml (deflated 39%) 2022-08-17T13:49:57.0204168Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131458.xml (deflated 39%) 2022-08-17T13:49:57.0204799Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131459.xml (deflated 39%) 2022-08-17T13:49:57.0205399Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131500.xml (deflated 39%) 2022-08-17T13:49:57.0206202Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131502.xml (deflated 39%) 2022-08-17T13:49:57.0206731Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131503.xml (deflated 39%) 2022-08-17T13:49:57.0207344Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131504.xml (deflated 39%) 2022-08-17T13:49:57.0207969Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131506.xml (deflated 39%) 2022-08-17T13:49:57.0208636Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131507.xml (deflated 39%) 2022-08-17T13:49:57.0209291Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131508.xml (deflated 39%) 2022-08-17T13:49:57.0209918Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220817131510.xml (deflated 40%) 2022-08-17T13:49:57.0210632Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131511.xml (deflated 42%) 2022-08-17T13:49:57.0211413Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131512.xml (deflated 42%) 2022-08-17T13:49:57.0212177Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131514.xml (deflated 42%) 2022-08-17T13:49:57.0212918Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131515.xml (deflated 41%) 2022-08-17T13:49:57.0213716Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131516.xml (deflated 42%) 2022-08-17T13:49:57.0214590Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131518.xml (deflated 43%) 2022-08-17T13:49:57.0215367Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131519.xml (deflated 42%) 2022-08-17T13:49:57.0216187Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131520.xml (deflated 42%) 2022-08-17T13:49:57.0216968Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131522.xml (deflated 43%) 2022-08-17T13:49:57.0217789Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131523.xml (deflated 44%) 2022-08-17T13:49:57.0218632Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131526.xml (deflated 45%) 2022-08-17T13:49:57.0219404Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131529.xml (deflated 43%) 2022-08-17T13:49:57.0220188Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131533.xml (deflated 42%) 2022-08-17T13:49:57.0221004Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131536.xml (deflated 45%) 2022-08-17T13:49:57.0221782Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131539.xml (deflated 45%) 2022-08-17T13:49:57.0222551Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131542.xml (deflated 46%) 2022-08-17T13:49:57.0223436Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131545.xml (deflated 46%) 2022-08-17T13:49:57.0224237Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131548.xml (deflated 44%) 2022-08-17T13:49:57.0225017Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131551.xml (deflated 45%) 2022-08-17T13:49:57.0225847Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131555.xml (deflated 45%) 2022-08-17T13:49:57.0226610Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131558.xml (deflated 44%) 2022-08-17T13:49:57.0227394Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131601.xml (deflated 44%) 2022-08-17T13:49:57.0228204Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131604.xml (deflated 43%) 2022-08-17T13:49:57.0228978Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131605.xml (deflated 42%) 2022-08-17T13:49:57.0229802Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131607.xml (deflated 42%) 2022-08-17T13:49:57.0230565Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131608.xml (deflated 44%) 2022-08-17T13:49:57.0231262Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131609.xml (deflated 44%) 2022-08-17T13:49:57.0231978Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131611.xml (deflated 42%) 2022-08-17T13:49:57.0232601Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131612.xml (deflated 42%) 2022-08-17T13:49:57.0233225Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131613.xml (deflated 42%) 2022-08-17T13:49:57.0233909Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131615.xml (deflated 43%) 2022-08-17T13:49:57.0234543Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131616.xml (deflated 42%) 2022-08-17T13:49:57.0235172Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131617.xml (deflated 43%) 2022-08-17T13:49:57.0235828Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131619.xml (deflated 42%) 2022-08-17T13:49:57.0236454Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131620.xml (deflated 42%) 2022-08-17T13:49:57.0237080Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131621.xml (deflated 41%) 2022-08-17T13:49:57.0237719Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131623.xml (deflated 41%) 2022-08-17T13:49:57.0238333Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131624.xml (deflated 41%) 2022-08-17T13:49:57.0238961Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131625.xml (deflated 41%) 2022-08-17T13:49:57.0239583Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131627.xml (deflated 43%) 2022-08-17T13:49:57.0240209Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131628.xml (deflated 42%) 2022-08-17T13:49:57.0240827Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131629.xml (deflated 43%) 2022-08-17T13:49:57.0241456Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131631.xml (deflated 42%) 2022-08-17T13:49:57.0242260Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131632.xml (deflated 42%) 2022-08-17T13:49:57.0243002Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131633.xml (deflated 42%) 2022-08-17T13:49:57.0243617Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131635.xml (deflated 41%) 2022-08-17T13:49:57.0244235Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131638.xml (deflated 43%) 2022-08-17T13:49:57.0244853Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131639.xml (deflated 42%) 2022-08-17T13:49:57.0245484Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131640.xml (deflated 43%) 2022-08-17T13:49:57.0246203Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131642.xml (deflated 43%) 2022-08-17T13:49:57.0246820Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131643.xml (deflated 42%) 2022-08-17T13:49:57.0247447Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131644.xml (deflated 42%) 2022-08-17T13:49:57.0248069Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131646.xml (deflated 43%) 2022-08-17T13:49:57.0248693Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131647.xml (deflated 43%) 2022-08-17T13:49:57.0249377Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131648.xml (deflated 43%) 2022-08-17T13:49:57.0250005Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131650.xml (deflated 43%) 2022-08-17T13:49:57.0250631Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131651.xml (deflated 42%) 2022-08-17T13:49:57.0251251Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131652.xml (deflated 44%) 2022-08-17T13:49:57.0251904Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131654.xml (deflated 42%) 2022-08-17T13:49:57.0252528Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131655.xml (deflated 43%) 2022-08-17T13:49:57.0253153Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131656.xml (deflated 42%) 2022-08-17T13:49:57.0253778Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131658.xml (deflated 41%) 2022-08-17T13:49:57.0254383Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131659.xml (deflated 42%) 2022-08-17T13:49:57.0254999Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220817131702.xml (deflated 42%) 2022-08-17T13:49:57.0255615Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131705.xml (deflated 41%) 2022-08-17T13:49:57.0256216Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131707.xml (deflated 41%) 2022-08-17T13:49:57.0256797Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131708.xml (deflated 42%) 2022-08-17T13:49:57.0257401Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131709.xml (deflated 41%) 2022-08-17T13:49:57.0258003Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131711.xml (deflated 41%) 2022-08-17T13:49:57.0258601Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131712.xml (deflated 41%) 2022-08-17T13:49:57.0259181Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131713.xml (deflated 42%) 2022-08-17T13:49:57.0259771Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131715.xml (deflated 42%) 2022-08-17T13:49:57.0260367Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220817131716.xml (deflated 42%) 2022-08-17T13:49:57.0260980Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220817131717.xml (deflated 41%) 2022-08-17T13:49:57.0261579Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131719.xml (deflated 41%) 2022-08-17T13:49:57.0262169Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131720.xml (deflated 41%) 2022-08-17T13:49:57.0262839Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131721.xml (deflated 42%) 2022-08-17T13:49:57.0263442Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131723.xml (deflated 42%) 2022-08-17T13:49:57.0264021Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131724.xml (deflated 41%) 2022-08-17T13:49:57.0264653Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131725.xml (deflated 41%) 2022-08-17T13:49:57.0265252Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131727.xml (deflated 42%) 2022-08-17T13:49:57.0265842Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131728.xml (deflated 41%) 2022-08-17T13:49:57.0266418Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131729.xml (deflated 41%) 2022-08-17T13:49:57.0267046Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131731.xml (deflated 41%) 2022-08-17T13:49:57.0267638Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131732.xml (deflated 41%) 2022-08-17T13:49:57.0268228Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131733.xml (deflated 41%) 2022-08-17T13:49:57.0268810Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131735.xml (deflated 41%) 2022-08-17T13:49:57.0269403Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131736.xml (deflated 41%) 2022-08-17T13:49:57.0269996Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131737.xml (deflated 42%) 2022-08-17T13:49:57.0270585Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131739.xml (deflated 41%) 2022-08-17T13:49:57.0271164Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131740.xml (deflated 41%) 2022-08-17T13:49:57.0271753Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220817131741.xml (deflated 41%) 2022-08-17T13:49:57.0272341Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220817131743.xml (deflated 41%) 2022-08-17T13:49:57.0272899Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220817131744.xml (deflated 41%) 2022-08-17T13:49:57.0273488Z adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220817131746.xml (deflated 69%) 2022-08-17T13:49:57.0274117Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20220817131753.xml (deflated 86%) 2022-08-17T13:49:57.0274738Z adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20220817131753.xml (deflated 86%) 2022-08-17T13:49:57.0275439Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131848.xml (deflated 44%) 2022-08-17T13:49:57.0276187Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131850.xml (deflated 44%) 2022-08-17T13:49:57.0276936Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220817131851.xml (deflated 44%) 2022-08-17T13:49:57.0277649Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131853.xml (deflated 41%) 2022-08-17T13:49:57.0278312Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131855.xml (deflated 42%) 2022-08-17T13:49:57.0278959Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131856.xml (deflated 42%) 2022-08-17T13:49:57.0279618Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220817131858.xml (deflated 42%) 2022-08-17T13:49:57.0280337Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131900.xml (deflated 43%) 2022-08-17T13:49:57.0281016Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131903.xml (deflated 44%) 2022-08-17T13:49:57.0281690Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131907.xml (deflated 43%) 2022-08-17T13:49:57.0282348Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131910.xml (deflated 43%) 2022-08-17T13:49:57.0283047Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131913.xml (deflated 43%) 2022-08-17T13:49:57.0283725Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131917.xml (deflated 43%) 2022-08-17T13:49:57.0284398Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131920.xml (deflated 43%) 2022-08-17T13:49:57.0285059Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220817131924.xml (deflated 43%) 2022-08-17T13:49:57.0285721Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131929.xml (deflated 42%) 2022-08-17T13:49:57.0286471Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131931.xml (deflated 42%) 2022-08-17T13:49:57.0287127Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131932.xml (deflated 42%) 2022-08-17T13:49:57.0287770Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220817131934.xml (deflated 41%) 2022-08-17T13:49:57.0288433Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131936.xml (deflated 43%) 2022-08-17T13:49:57.0289107Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131937.xml (deflated 43%) 2022-08-17T13:49:57.0289775Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131939.xml (deflated 43%) 2022-08-17T13:49:57.0290444Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131941.xml (deflated 43%) 2022-08-17T13:49:57.0291111Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131942.xml (deflated 43%) 2022-08-17T13:49:57.0291782Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131944.xml (deflated 43%) 2022-08-17T13:49:57.0292455Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131946.xml (deflated 43%) 2022-08-17T13:49:57.0293107Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131947.xml (deflated 43%) 2022-08-17T13:49:57.0293772Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220817131949.xml (deflated 42%) 2022-08-17T13:49:57.0294397Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220817131951.xml (deflated 90%) 2022-08-17T13:49:57.0295030Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220817131951.xml (deflated 91%) 2022-08-17T13:49:57.0295658Z adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220817134812.xml (deflated 43%) 2022-08-17T13:49:57.0296331Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134816.xml (deflated 40%) 2022-08-17T13:49:57.0296985Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134820.xml (deflated 41%) 2022-08-17T13:49:57.0297634Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134824.xml (deflated 41%) 2022-08-17T13:49:57.0298308Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134827.xml (deflated 41%) 2022-08-17T13:49:57.0298944Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134830.xml (deflated 41%) 2022-08-17T13:49:57.0299592Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134834.xml (deflated 41%) 2022-08-17T13:49:57.0300239Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134837.xml (deflated 41%) 2022-08-17T13:49:57.0300881Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134841.xml (deflated 41%) 2022-08-17T13:49:57.0301512Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220817134844.xml (deflated 40%) 2022-08-17T13:49:57.0302161Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134848.xml (deflated 41%) 2022-08-17T13:49:57.0302876Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134849.xml (deflated 41%) 2022-08-17T13:49:57.0303521Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134850.xml (deflated 41%) 2022-08-17T13:49:57.0304158Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134852.xml (deflated 41%) 2022-08-17T13:49:57.0304805Z adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220817134853.xml (deflated 41%) 2022-08-17T13:49:57.0305395Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817134855.xml (deflated 40%) 2022-08-17T13:49:57.0305942Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220817134857.xml (deflated 40%) 2022-08-17T13:49:57.0306471Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817134859.xml (deflated 40%) 2022-08-17T13:49:57.0307013Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220817134900.xml (deflated 40%) 2022-08-17T13:49:57.0307579Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817134902.xml (deflated 40%) 2022-08-17T13:49:57.0308156Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220817134904.xml (deflated 40%) 2022-08-17T13:49:57.0308721Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817134905.xml (deflated 39%) 2022-08-17T13:49:57.0309295Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220817134907.xml (deflated 39%) 2022-08-17T13:49:57.0309855Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220817134909.xml (deflated 40%) 2022-08-17T13:49:57.0310417Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220817134910.xml (deflated 40%) 2022-08-17T13:49:57.0310978Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817134912.xml (deflated 40%) 2022-08-17T13:49:57.0311586Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220817134914.xml (deflated 40%) 2022-08-17T13:49:57.0312144Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134915.xml (deflated 39%) 2022-08-17T13:49:57.0312702Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134917.xml (deflated 40%) 2022-08-17T13:49:57.0313245Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134919.xml (deflated 39%) 2022-08-17T13:49:57.0313842Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220817134920.xml (deflated 40%) 2022-08-17T13:49:57.0314396Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817134932.xml (deflated 39%) 2022-08-17T13:49:57.0314943Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220817134934.xml (deflated 39%) 2022-08-17T13:49:57.0315476Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134936.xml (deflated 39%) 2022-08-17T13:49:57.0316010Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134937.xml (deflated 39%) 2022-08-17T13:49:57.0316544Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134939.xml (deflated 38%) 2022-08-17T13:49:57.0317079Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134941.xml (deflated 38%) 2022-08-17T13:49:57.0317604Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134942.xml (deflated 38%) 2022-08-17T13:49:57.0318132Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134944.xml (deflated 39%) 2022-08-17T13:49:57.0318664Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134946.xml (deflated 39%) 2022-08-17T13:49:57.0319182Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220817134949.xml (deflated 39%) 2022-08-17T13:49:57.0319811Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121142.xml (deflated 42%) 2022-08-17T13:49:57.0320527Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121145.xml (deflated 42%) 2022-08-17T13:49:57.0321230Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121148.xml (deflated 42%) 2022-08-17T13:49:57.0321935Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121149.xml (deflated 42%) 2022-08-17T13:49:57.0322635Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121151.xml (deflated 42%) 2022-08-17T13:49:57.0323336Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121152.xml (deflated 42%) 2022-08-17T13:49:57.0324033Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121155.xml (deflated 41%) 2022-08-17T13:49:57.0324729Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121158.xml (deflated 42%) 2022-08-17T13:49:57.0325419Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121201.xml (deflated 43%) 2022-08-17T13:49:57.0326202Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121202.xml (deflated 43%) 2022-08-17T13:49:57.0326956Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121204.xml (deflated 42%) 2022-08-17T13:49:57.0327651Z adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220817121205.xml (deflated 42%) 2022-08-17T13:49:57.0328298Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817131957.xml (deflated 42%) 2022-08-17T13:49:57.0328925Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132000.xml (deflated 42%) 2022-08-17T13:49:57.0329603Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132002.xml (deflated 42%) 2022-08-17T13:49:57.0330230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132005.xml (deflated 42%) 2022-08-17T13:49:57.0330839Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132009.xml (deflated 41%) 2022-08-17T13:49:57.0331464Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132013.xml (deflated 41%) 2022-08-17T13:49:57.0332086Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132017.xml (deflated 42%) 2022-08-17T13:49:57.0332699Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132021.xml (deflated 41%) 2022-08-17T13:49:57.0333576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132024.xml (deflated 42%) 2022-08-17T13:49:57.0334184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132028.xml (deflated 41%) 2022-08-17T13:49:57.0334804Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132031.xml (deflated 41%) 2022-08-17T13:49:57.0335417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132035.xml (deflated 41%) 2022-08-17T13:49:57.0336028Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132039.xml (deflated 41%) 2022-08-17T13:49:57.0336633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132042.xml (deflated 42%) 2022-08-17T13:49:57.0337247Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132046.xml (deflated 41%) 2022-08-17T13:49:57.0337854Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132050.xml (deflated 42%) 2022-08-17T13:49:57.0338463Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132054.xml (deflated 43%) 2022-08-17T13:49:57.0339063Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132057.xml (deflated 42%) 2022-08-17T13:49:57.0339671Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132059.xml (deflated 46%) 2022-08-17T13:49:57.0340279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132102.xml (deflated 47%) 2022-08-17T13:49:57.0340885Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132106.xml (deflated 48%) 2022-08-17T13:49:57.0341489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132110.xml (deflated 46%) 2022-08-17T13:49:57.0342101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132114.xml (deflated 41%) 2022-08-17T13:49:57.0342805Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132117.xml (deflated 41%) 2022-08-17T13:49:57.0343423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132121.xml (deflated 41%) 2022-08-17T13:49:57.0344021Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132125.xml (deflated 41%) 2022-08-17T13:49:57.0344659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132129.xml (deflated 41%) 2022-08-17T13:49:57.0345274Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132133.xml (deflated 41%) 2022-08-17T13:49:57.0345885Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132136.xml (deflated 41%) 2022-08-17T13:49:57.0346490Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132140.xml (deflated 42%) 2022-08-17T13:49:57.0347106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132142.xml (deflated 41%) 2022-08-17T13:49:57.0347716Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132143.xml (deflated 40%) 2022-08-17T13:49:57.0348326Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132147.xml (deflated 41%) 2022-08-17T13:49:57.0348931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132151.xml (deflated 43%) 2022-08-17T13:49:57.0349535Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132152.xml (deflated 43%) 2022-08-17T13:49:57.0350146Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132154.xml (deflated 40%) 2022-08-17T13:49:57.0350758Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132158.xml (deflated 40%) 2022-08-17T13:49:57.0351357Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132201.xml (deflated 41%) 2022-08-17T13:49:57.0351972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132205.xml (deflated 41%) 2022-08-17T13:49:57.0352580Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132209.xml (deflated 41%) 2022-08-17T13:49:57.0353193Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132213.xml (deflated 41%) 2022-08-17T13:49:57.0353793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132217.xml (deflated 40%) 2022-08-17T13:49:57.0354396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132221.xml (deflated 41%) 2022-08-17T13:49:57.0355009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132224.xml (deflated 41%) 2022-08-17T13:49:57.0355615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132228.xml (deflated 41%) 2022-08-17T13:49:57.0355962Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132232.xml (deflated 41%) 2022-08-17T13:49:57.0356297Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132236.xml (deflated 41%) 2022-08-17T13:49:57.0356689Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132239.xml (deflated 41%) 2022-08-17T13:49:57.0357032Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132243.xml (deflated 41%) 2022-08-17T13:49:57.0357374Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132247.xml (deflated 41%) 2022-08-17T13:49:57.0357719Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132251.xml (deflated 41%) 2022-08-17T13:49:57.0358084Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132254.xml (deflated 40%) 2022-08-17T13:49:57.0358428Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132258.xml (deflated 41%) 2022-08-17T13:49:57.0358767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132302.xml (deflated 41%) 2022-08-17T13:49:57.0359112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132306.xml (deflated 40%) 2022-08-17T13:49:57.0359450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132310.xml (deflated 41%) 2022-08-17T13:49:57.0359781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132314.xml (deflated 41%) 2022-08-17T13:49:57.0360123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132318.xml (deflated 41%) 2022-08-17T13:49:57.0360466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132321.xml (deflated 41%) 2022-08-17T13:49:57.0360807Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132325.xml (deflated 41%) 2022-08-17T13:49:57.0361152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132329.xml (deflated 41%) 2022-08-17T13:49:57.0361494Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132333.xml (deflated 42%) 2022-08-17T13:49:57.0361832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132337.xml (deflated 41%) 2022-08-17T13:49:57.0362178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132341.xml (deflated 41%) 2022-08-17T13:49:57.0362519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132344.xml (deflated 41%) 2022-08-17T13:49:57.0362861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132348.xml (deflated 41%) 2022-08-17T13:49:57.0363202Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132352.xml (deflated 40%) 2022-08-17T13:49:57.0363529Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132355.xml (deflated 41%) 2022-08-17T13:49:57.0363869Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132359.xml (deflated 42%) 2022-08-17T13:49:57.0364213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132403.xml (deflated 41%) 2022-08-17T13:49:57.0364559Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132407.xml (deflated 41%) 2022-08-17T13:49:57.0364901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132410.xml (deflated 43%) 2022-08-17T13:49:57.0365267Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132412.xml (deflated 42%) 2022-08-17T13:49:57.0365608Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132413.xml (deflated 42%) 2022-08-17T13:49:57.0366042Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132415.xml (deflated 42%) 2022-08-17T13:49:57.0366382Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132416.xml (deflated 42%) 2022-08-17T13:49:57.0366763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132418.xml (deflated 42%) 2022-08-17T13:49:57.0367097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132419.xml (deflated 42%) 2022-08-17T13:49:57.0367440Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132421.xml (deflated 42%) 2022-08-17T13:49:57.0367782Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132422.xml (deflated 43%) 2022-08-17T13:49:57.0368123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132424.xml (deflated 43%) 2022-08-17T13:49:57.0368469Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132425.xml (deflated 43%) 2022-08-17T13:49:57.0368813Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132427.xml (deflated 43%) 2022-08-17T13:49:57.0369156Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132428.xml (deflated 42%) 2022-08-17T13:49:57.0369499Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132430.xml (deflated 43%) 2022-08-17T13:49:57.0369844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132431.xml (deflated 42%) 2022-08-17T13:49:57.0370185Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132433.xml (deflated 43%) 2022-08-17T13:49:57.0370525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132434.xml (deflated 42%) 2022-08-17T13:49:57.0370861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132436.xml (deflated 42%) 2022-08-17T13:49:57.0371202Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132437.xml (deflated 43%) 2022-08-17T13:49:57.0371545Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132439.xml (deflated 42%) 2022-08-17T13:49:57.0371888Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132440.xml (deflated 42%) 2022-08-17T13:49:57.0372231Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132442.xml (deflated 43%) 2022-08-17T13:49:57.0372568Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132443.xml (deflated 42%) 2022-08-17T13:49:57.0372911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132445.xml (deflated 42%) 2022-08-17T13:49:57.0373252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132446.xml (deflated 42%) 2022-08-17T13:49:57.0373594Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132450.xml (deflated 42%) 2022-08-17T13:49:57.0373985Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132451.xml (deflated 42%) 2022-08-17T13:49:57.0374317Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132453.xml (deflated 41%) 2022-08-17T13:49:57.0374656Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132458.xml (deflated 42%) 2022-08-17T13:49:57.0375023Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132502.xml (deflated 41%) 2022-08-17T13:49:57.0375366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132507.xml (deflated 41%) 2022-08-17T13:49:57.0375708Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132510.xml (deflated 41%) 2022-08-17T13:49:57.0376051Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132515.xml (deflated 41%) 2022-08-17T13:49:57.0376390Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132519.xml (deflated 40%) 2022-08-17T13:49:57.0376732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132523.xml (deflated 42%) 2022-08-17T13:49:57.0377072Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132525.xml (deflated 40%) 2022-08-17T13:49:57.0377416Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132533.xml (deflated 40%) 2022-08-17T13:49:57.0377755Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132537.xml (deflated 41%) 2022-08-17T13:49:57.0378089Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132541.xml (deflated 42%) 2022-08-17T13:49:57.0378430Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132542.xml (deflated 41%) 2022-08-17T13:49:57.0378774Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132544.xml (deflated 41%) 2022-08-17T13:49:57.0379112Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132545.xml (deflated 41%) 2022-08-17T13:49:57.0379457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132547.xml (deflated 41%) 2022-08-17T13:49:57.0379797Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132548.xml (deflated 41%) 2022-08-17T13:49:57.0380141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132550.xml (deflated 41%) 2022-08-17T13:49:57.0380481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132551.xml (deflated 41%) 2022-08-17T13:49:57.0380822Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132553.xml (deflated 41%) 2022-08-17T13:49:57.0381164Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132557.xml (deflated 42%) 2022-08-17T13:49:57.0381497Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132559.xml (deflated 40%) 2022-08-17T13:49:57.0381837Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132602.xml (deflated 40%) 2022-08-17T13:49:57.0382210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132606.xml (deflated 41%) 2022-08-17T13:49:57.0382547Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132610.xml (deflated 42%) 2022-08-17T13:49:57.0382957Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132612.xml (deflated 41%) 2022-08-17T13:49:57.0383297Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132613.xml (deflated 41%) 2022-08-17T13:49:57.0383672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132615.xml (deflated 42%) 2022-08-17T13:49:57.0384016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132619.xml (deflated 41%) 2022-08-17T13:49:57.0384354Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132622.xml (deflated 41%) 2022-08-17T13:49:57.0384697Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132626.xml (deflated 42%) 2022-08-17T13:49:57.0385026Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132630.xml (deflated 41%) 2022-08-17T13:49:57.0385367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132631.xml (deflated 41%) 2022-08-17T13:49:57.0385709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132635.xml (deflated 41%) 2022-08-17T13:49:57.0386050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132639.xml (deflated 41%) 2022-08-17T13:49:57.0386394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132640.xml (deflated 41%) 2022-08-17T13:49:57.0386741Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132642.xml (deflated 41%) 2022-08-17T13:49:57.0387083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132645.xml (deflated 41%) 2022-08-17T13:49:57.0387422Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132647.xml (deflated 41%) 2022-08-17T13:49:57.0387764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132648.xml (deflated 42%) 2022-08-17T13:49:57.0388103Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132652.xml (deflated 41%) 2022-08-17T13:49:57.0388443Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132656.xml (deflated 41%) 2022-08-17T13:49:57.0388775Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132659.xml (deflated 41%) 2022-08-17T13:49:57.0389116Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132703.xml (deflated 41%) 2022-08-17T13:49:57.0389459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132707.xml (deflated 41%) 2022-08-17T13:49:57.0389801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132710.xml (deflated 42%) 2022-08-17T13:49:57.0390143Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132714.xml (deflated 42%) 2022-08-17T13:49:57.0390482Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132718.xml (deflated 42%) 2022-08-17T13:49:57.0390855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132721.xml (deflated 42%) 2022-08-17T13:49:57.0391196Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132725.xml (deflated 41%) 2022-08-17T13:49:57.0391535Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132729.xml (deflated 42%) 2022-08-17T13:49:57.0391878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132732.xml (deflated 42%) 2022-08-17T13:49:57.0392230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132736.xml (deflated 42%) 2022-08-17T13:49:57.0392570Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132740.xml (deflated 42%) 2022-08-17T13:49:57.0392913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132743.xml (deflated 42%) 2022-08-17T13:49:57.0393254Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132747.xml (deflated 42%) 2022-08-17T13:49:57.0393593Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132751.xml (deflated 42%) 2022-08-17T13:49:57.0393932Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132754.xml (deflated 41%) 2022-08-17T13:49:57.0394273Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132756.xml (deflated 42%) 2022-08-17T13:49:57.0394616Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132759.xml (deflated 41%) 2022-08-17T13:49:57.0394959Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132803.xml (deflated 40%) 2022-08-17T13:49:57.0395300Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132807.xml (deflated 42%) 2022-08-17T13:49:57.0395641Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132811.xml (deflated 41%) 2022-08-17T13:49:57.0395969Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132812.xml (deflated 41%) 2022-08-17T13:49:57.0396312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132814.xml (deflated 42%) 2022-08-17T13:49:57.0396652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132815.xml (deflated 42%) 2022-08-17T13:49:57.0396995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132817.xml (deflated 42%) 2022-08-17T13:49:57.0397338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132818.xml (deflated 42%) 2022-08-17T13:49:57.0397679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132822.xml (deflated 42%) 2022-08-17T13:49:57.0398017Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132825.xml (deflated 42%) 2022-08-17T13:49:57.0398359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132827.xml (deflated 42%) 2022-08-17T13:49:57.0398699Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132828.xml (deflated 42%) 2022-08-17T13:49:57.0399036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132832.xml (deflated 42%) 2022-08-17T13:49:57.0399394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132836.xml (deflated 41%) 2022-08-17T13:49:57.0399735Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132837.xml (deflated 42%) 2022-08-17T13:49:57.0400074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132841.xml (deflated 41%) 2022-08-17T13:49:57.0400450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132845.xml (deflated 42%) 2022-08-17T13:49:57.0400796Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132848.xml (deflated 42%) 2022-08-17T13:49:57.0401138Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132852.xml (deflated 42%) 2022-08-17T13:49:57.0401480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132856.xml (deflated 41%) 2022-08-17T13:49:57.0401823Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132859.xml (deflated 42%) 2022-08-17T13:49:57.0402162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132901.xml (deflated 41%) 2022-08-17T13:49:57.0402503Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132903.xml (deflated 41%) 2022-08-17T13:49:57.0402836Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132907.xml (deflated 41%) 2022-08-17T13:49:57.0403175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132911.xml (deflated 42%) 2022-08-17T13:49:57.0403519Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132914.xml (deflated 41%) 2022-08-17T13:49:57.0403858Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132916.xml (deflated 40%) 2022-08-17T13:49:57.0404199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132919.xml (deflated 41%) 2022-08-17T13:49:57.0404538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132923.xml (deflated 41%) 2022-08-17T13:49:57.0404880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132927.xml (deflated 43%) 2022-08-17T13:49:57.0405220Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132928.xml (deflated 41%) 2022-08-17T13:49:57.0405563Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132932.xml (deflated 41%) 2022-08-17T13:49:57.0405989Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132936.xml (deflated 40%) 2022-08-17T13:49:57.0406331Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132940.xml (deflated 42%) 2022-08-17T13:49:57.0406661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132942.xml (deflated 41%) 2022-08-17T13:49:57.0407004Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132946.xml (deflated 40%) 2022-08-17T13:49:57.0407345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132949.xml (deflated 41%) 2022-08-17T13:49:57.0407733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132953.xml (deflated 40%) 2022-08-17T13:49:57.0408078Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817132957.xml (deflated 40%) 2022-08-17T13:49:57.0408421Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133001.xml (deflated 41%) 2022-08-17T13:49:57.0408764Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133003.xml (deflated 41%) 2022-08-17T13:49:57.0409132Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133006.xml (deflated 41%) 2022-08-17T13:49:57.0409476Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133010.xml (deflated 41%) 2022-08-17T13:49:57.0409817Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133014.xml (deflated 41%) 2022-08-17T13:49:57.0410148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133018.xml (deflated 41%) 2022-08-17T13:49:57.0410488Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133019.xml (deflated 41%) 2022-08-17T13:49:57.0410831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133021.xml (deflated 41%) 2022-08-17T13:49:57.0411173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133027.xml (deflated 41%) 2022-08-17T13:49:57.0411514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133032.xml (deflated 40%) 2022-08-17T13:49:57.0411855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133036.xml (deflated 40%) 2022-08-17T13:49:57.0412198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133040.xml (deflated 41%) 2022-08-17T13:49:57.0412541Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133044.xml (deflated 42%) 2022-08-17T13:49:57.0412886Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133046.xml (deflated 42%) 2022-08-17T13:49:57.0413234Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133047.xml (deflated 42%) 2022-08-17T13:49:57.0413576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133049.xml (deflated 42%) 2022-08-17T13:49:57.0413904Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133050.xml (deflated 43%) 2022-08-17T13:49:57.0414248Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133052.xml (deflated 42%) 2022-08-17T13:49:57.0414591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133053.xml (deflated 42%) 2022-08-17T13:49:57.0414936Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133055.xml (deflated 42%) 2022-08-17T13:49:57.0415277Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133056.xml (deflated 41%) 2022-08-17T13:49:57.0415618Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133100.xml (deflated 42%) 2022-08-17T13:49:57.0415957Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133104.xml (deflated 42%) 2022-08-17T13:49:57.0416324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133105.xml (deflated 43%) 2022-08-17T13:49:57.0416665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133107.xml (deflated 42%) 2022-08-17T13:49:57.0417005Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133109.xml (deflated 41%) 2022-08-17T13:49:57.0417336Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133112.xml (deflated 42%) 2022-08-17T13:49:57.0417702Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133116.xml (deflated 41%) 2022-08-17T13:49:57.0418043Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133119.xml (deflated 41%) 2022-08-17T13:49:57.0418387Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133123.xml (deflated 41%) 2022-08-17T13:49:57.0418726Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133127.xml (deflated 41%) 2022-08-17T13:49:57.0419065Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133130.xml (deflated 41%) 2022-08-17T13:49:57.0419404Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133134.xml (deflated 41%) 2022-08-17T13:49:57.0419747Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133138.xml (deflated 41%) 2022-08-17T13:49:57.0420086Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133142.xml (deflated 41%) 2022-08-17T13:49:57.0420427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133145.xml (deflated 40%) 2022-08-17T13:49:57.0420767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133149.xml (deflated 40%) 2022-08-17T13:49:57.0421098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133153.xml (deflated 41%) 2022-08-17T13:49:57.0421438Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133157.xml (deflated 41%) 2022-08-17T13:49:57.0421780Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133201.xml (deflated 41%) 2022-08-17T13:49:57.0422119Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133205.xml (deflated 41%) 2022-08-17T13:49:57.0422461Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133208.xml (deflated 41%) 2022-08-17T13:49:57.0422880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133212.xml (deflated 41%) 2022-08-17T13:49:57.0423222Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133216.xml (deflated 43%) 2022-08-17T13:49:57.0423563Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133218.xml (deflated 41%) 2022-08-17T13:49:57.0423906Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133221.xml (deflated 41%) 2022-08-17T13:49:57.0424245Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133225.xml (deflated 42%) 2022-08-17T13:49:57.0424574Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133227.xml (deflated 42%) 2022-08-17T13:49:57.0424947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133228.xml (deflated 41%) 2022-08-17T13:49:57.0425289Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133232.xml (deflated 41%) 2022-08-17T13:49:57.0425632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133236.xml (deflated 41%) 2022-08-17T13:49:57.0425995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133239.xml (deflated 41%) 2022-08-17T13:49:57.0426336Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133243.xml (deflated 41%) 2022-08-17T13:49:57.0426678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133245.xml (deflated 41%) 2022-08-17T13:49:57.0427019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133246.xml (deflated 40%) 2022-08-17T13:49:57.0427363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133250.xml (deflated 41%) 2022-08-17T13:49:57.0427703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133254.xml (deflated 40%) 2022-08-17T13:49:57.0428032Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133257.xml (deflated 41%) 2022-08-17T13:49:57.0428373Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133301.xml (deflated 41%) 2022-08-17T13:49:57.0428715Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133304.xml (deflated 41%) 2022-08-17T13:49:57.0429057Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133308.xml (deflated 41%) 2022-08-17T13:49:57.0429397Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133312.xml (deflated 40%) 2022-08-17T13:49:57.0429736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133316.xml (deflated 42%) 2022-08-17T13:49:57.0430072Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133317.xml (deflated 41%) 2022-08-17T13:49:57.0430415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133319.xml (deflated 41%) 2022-08-17T13:49:57.0430754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133320.xml (deflated 41%) 2022-08-17T13:49:57.0431096Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133324.xml (deflated 41%) 2022-08-17T13:49:57.0431439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133328.xml (deflated 41%) 2022-08-17T13:49:57.0431767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133332.xml (deflated 41%) 2022-08-17T13:49:57.0432107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133336.xml (deflated 41%) 2022-08-17T13:49:57.0432457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133339.xml (deflated 42%) 2022-08-17T13:49:57.0432802Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133343.xml (deflated 42%) 2022-08-17T13:49:57.0433177Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133347.xml (deflated 40%) 2022-08-17T13:49:57.0433517Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133350.xml (deflated 42%) 2022-08-17T13:49:57.0433860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133354.xml (deflated 42%) 2022-08-17T13:49:57.0434197Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133358.xml (deflated 41%) 2022-08-17T13:49:57.0434562Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133359.xml (deflated 41%) 2022-08-17T13:49:57.0434902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133402.xml (deflated 42%) 2022-08-17T13:49:57.0435231Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133406.xml (deflated 42%) 2022-08-17T13:49:57.0435573Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133408.xml (deflated 42%) 2022-08-17T13:49:57.0435914Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133411.xml (deflated 42%) 2022-08-17T13:49:57.0436257Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133415.xml (deflated 41%) 2022-08-17T13:49:57.0436599Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133419.xml (deflated 41%) 2022-08-17T13:49:57.0436941Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133423.xml (deflated 42%) 2022-08-17T13:49:57.0437282Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133426.xml (deflated 41%) 2022-08-17T13:49:57.0437623Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133430.xml (deflated 42%) 2022-08-17T13:49:57.0437959Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133434.xml (deflated 40%) 2022-08-17T13:49:57.0438302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133437.xml (deflated 41%) 2022-08-17T13:49:57.0438645Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133441.xml (deflated 41%) 2022-08-17T13:49:57.0438976Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133445.xml (deflated 41%) 2022-08-17T13:49:57.0439318Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133448.xml (deflated 42%) 2022-08-17T13:49:57.0439660Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133452.xml (deflated 41%) 2022-08-17T13:49:57.0440001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133456.xml (deflated 42%) 2022-08-17T13:49:57.0440342Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133459.xml (deflated 43%) 2022-08-17T13:49:57.0440680Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133503.xml (deflated 42%) 2022-08-17T13:49:57.0441022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133505.xml (deflated 46%) 2022-08-17T13:49:57.0441360Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133508.xml (deflated 47%) 2022-08-17T13:49:57.0441727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133512.xml (deflated 48%) 2022-08-17T13:49:57.0442072Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133516.xml (deflated 46%) 2022-08-17T13:49:57.0442401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133520.xml (deflated 41%) 2022-08-17T13:49:57.0442743Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133523.xml (deflated 40%) 2022-08-17T13:49:57.0443107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133527.xml (deflated 41%) 2022-08-17T13:49:57.0443449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133531.xml (deflated 41%) 2022-08-17T13:49:57.0443791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133535.xml (deflated 41%) 2022-08-17T13:49:57.0444129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133539.xml (deflated 41%) 2022-08-17T13:49:57.0444471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133542.xml (deflated 40%) 2022-08-17T13:49:57.0444817Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133546.xml (deflated 42%) 2022-08-17T13:49:57.0445159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133548.xml (deflated 41%) 2022-08-17T13:49:57.0445501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133549.xml (deflated 40%) 2022-08-17T13:49:57.0445832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133553.xml (deflated 41%) 2022-08-17T13:49:57.0446259Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133557.xml (deflated 43%) 2022-08-17T13:49:57.0446601Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133558.xml (deflated 43%) 2022-08-17T13:49:57.0446943Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133600.xml (deflated 40%) 2022-08-17T13:49:57.0447285Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133603.xml (deflated 40%) 2022-08-17T13:49:57.0447626Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133607.xml (deflated 40%) 2022-08-17T13:49:57.0447969Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133611.xml (deflated 41%) 2022-08-17T13:49:57.0448312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133615.xml (deflated 41%) 2022-08-17T13:49:57.0448652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133619.xml (deflated 41%) 2022-08-17T13:49:57.0448994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133622.xml (deflated 40%) 2022-08-17T13:49:57.0449338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133626.xml (deflated 41%) 2022-08-17T13:49:57.0449668Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133630.xml (deflated 41%) 2022-08-17T13:49:57.0450011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133634.xml (deflated 41%) 2022-08-17T13:49:57.0450396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133638.xml (deflated 41%) 2022-08-17T13:49:57.0450735Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133642.xml (deflated 41%) 2022-08-17T13:49:57.0451083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133645.xml (deflated 41%) 2022-08-17T13:49:57.0451423Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133649.xml (deflated 41%) 2022-08-17T13:49:57.0451791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133653.xml (deflated 41%) 2022-08-17T13:49:57.0452135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133656.xml (deflated 41%) 2022-08-17T13:49:57.0452479Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133700.xml (deflated 41%) 2022-08-17T13:49:57.0452818Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133704.xml (deflated 41%) 2022-08-17T13:49:57.0453148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133708.xml (deflated 41%) 2022-08-17T13:49:57.0453489Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133712.xml (deflated 40%) 2022-08-17T13:49:57.0453834Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133716.xml (deflated 40%) 2022-08-17T13:49:57.0454179Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133719.xml (deflated 41%) 2022-08-17T13:49:57.0454525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133723.xml (deflated 41%) 2022-08-17T13:49:57.0454870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133727.xml (deflated 41%) 2022-08-17T13:49:57.0455213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133731.xml (deflated 41%) 2022-08-17T13:49:57.0455556Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133735.xml (deflated 41%) 2022-08-17T13:49:57.0455905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133739.xml (deflated 42%) 2022-08-17T13:49:57.0456244Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133742.xml (deflated 41%) 2022-08-17T13:49:57.0456590Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133746.xml (deflated 41%) 2022-08-17T13:49:57.0456921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133750.xml (deflated 42%) 2022-08-17T13:49:57.0457265Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133754.xml (deflated 41%) 2022-08-17T13:49:57.0457607Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133757.xml (deflated 41%) 2022-08-17T13:49:57.0457955Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133801.xml (deflated 41%) 2022-08-17T13:49:57.0458296Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133805.xml (deflated 42%) 2022-08-17T13:49:57.0458665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133809.xml (deflated 41%) 2022-08-17T13:49:57.0459003Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133812.xml (deflated 41%) 2022-08-17T13:49:57.0459345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133816.xml (deflated 43%) 2022-08-17T13:49:57.0459686Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133818.xml (deflated 43%) 2022-08-17T13:49:57.0460050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133819.xml (deflated 43%) 2022-08-17T13:49:57.0460381Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133821.xml (deflated 42%) 2022-08-17T13:49:57.0460723Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133822.xml (deflated 42%) 2022-08-17T13:49:57.0461066Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133824.xml (deflated 42%) 2022-08-17T13:49:57.0461411Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133825.xml (deflated 42%) 2022-08-17T13:49:57.0461755Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133827.xml (deflated 42%) 2022-08-17T13:49:57.0462094Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133828.xml (deflated 43%) 2022-08-17T13:49:57.0462435Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133830.xml (deflated 43%) 2022-08-17T13:49:57.0462840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133831.xml (deflated 43%) 2022-08-17T13:49:57.0463187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133833.xml (deflated 43%) 2022-08-17T13:49:57.0463528Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133834.xml (deflated 42%) 2022-08-17T13:49:57.0463870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133836.xml (deflated 43%) 2022-08-17T13:49:57.0464200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133837.xml (deflated 42%) 2022-08-17T13:49:57.0464542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133839.xml (deflated 43%) 2022-08-17T13:49:57.0464882Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133840.xml (deflated 43%) 2022-08-17T13:49:57.0465226Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133842.xml (deflated 42%) 2022-08-17T13:49:57.0465569Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133843.xml (deflated 43%) 2022-08-17T13:49:57.0465909Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133845.xml (deflated 42%) 2022-08-17T13:49:57.0466250Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133846.xml (deflated 42%) 2022-08-17T13:49:57.0466595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133848.xml (deflated 43%) 2022-08-17T13:49:57.0466935Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133849.xml (deflated 42%) 2022-08-17T13:49:57.0467306Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133851.xml (deflated 43%) 2022-08-17T13:49:57.0467632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133852.xml (deflated 42%) 2022-08-17T13:49:57.0467971Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133856.xml (deflated 42%) 2022-08-17T13:49:57.0468313Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133857.xml (deflated 42%) 2022-08-17T13:49:57.0468679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133859.xml (deflated 41%) 2022-08-17T13:49:57.0469022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133904.xml (deflated 42%) 2022-08-17T13:49:57.0469365Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133907.xml (deflated 40%) 2022-08-17T13:49:57.0469707Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133912.xml (deflated 42%) 2022-08-17T13:49:57.0470045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133916.xml (deflated 41%) 2022-08-17T13:49:57.0470433Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133921.xml (deflated 41%) 2022-08-17T13:49:57.0470968Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133924.xml (deflated 40%) 2022-08-17T13:49:57.0471411Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133929.xml (deflated 41%) 2022-08-17T13:49:57.0471761Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133934.xml (deflated 41%) 2022-08-17T13:49:57.0472105Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133943.xml (deflated 40%) 2022-08-17T13:49:57.0472447Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133946.xml (deflated 40%) 2022-08-17T13:49:57.0472794Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133950.xml (deflated 42%) 2022-08-17T13:49:57.0473135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133952.xml (deflated 41%) 2022-08-17T13:49:57.0473474Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133953.xml (deflated 41%) 2022-08-17T13:49:57.0473814Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133955.xml (deflated 41%) 2022-08-17T13:49:57.0474159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133956.xml (deflated 41%) 2022-08-17T13:49:57.0474501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133958.xml (deflated 41%) 2022-08-17T13:49:57.0474843Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817133959.xml (deflated 41%) 2022-08-17T13:49:57.0475174Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134001.xml (deflated 41%) 2022-08-17T13:49:57.0475513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134002.xml (deflated 41%) 2022-08-17T13:49:57.0475856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134006.xml (deflated 42%) 2022-08-17T13:49:57.0476262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134008.xml (deflated 40%) 2022-08-17T13:49:57.0476608Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134012.xml (deflated 40%) 2022-08-17T13:49:57.0476947Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134016.xml (deflated 42%) 2022-08-17T13:49:57.0477290Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134019.xml (deflated 42%) 2022-08-17T13:49:57.0477654Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134021.xml (deflated 41%) 2022-08-17T13:49:57.0477998Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134023.xml (deflated 41%) 2022-08-17T13:49:57.0478341Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134024.xml (deflated 42%) 2022-08-17T13:49:57.0478670Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134028.xml (deflated 41%) 2022-08-17T13:49:57.0479010Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134032.xml (deflated 41%) 2022-08-17T13:49:57.0479350Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134035.xml (deflated 41%) 2022-08-17T13:49:57.0479696Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134039.xml (deflated 41%) 2022-08-17T13:49:57.0480039Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134041.xml (deflated 41%) 2022-08-17T13:49:57.0480386Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134044.xml (deflated 41%) 2022-08-17T13:49:57.0480725Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134048.xml (deflated 41%) 2022-08-17T13:49:57.0481066Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134049.xml (deflated 41%) 2022-08-17T13:49:57.0481406Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134051.xml (deflated 41%) 2022-08-17T13:49:57.0481752Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134055.xml (deflated 41%) 2022-08-17T13:49:57.0482093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134056.xml (deflated 41%) 2022-08-17T13:49:57.0482426Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134058.xml (deflated 42%) 2022-08-17T13:49:57.0482766Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134101.xml (deflated 42%) 2022-08-17T13:49:57.0483106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134105.xml (deflated 41%) 2022-08-17T13:49:57.0483449Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134109.xml (deflated 41%) 2022-08-17T13:49:57.0483793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134112.xml (deflated 41%) 2022-08-17T13:49:57.0484135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134116.xml (deflated 41%) 2022-08-17T13:49:57.0484477Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134120.xml (deflated 42%) 2022-08-17T13:49:57.0484846Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134123.xml (deflated 42%) 2022-08-17T13:49:57.0485189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134127.xml (deflated 42%) 2022-08-17T13:49:57.0485533Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134131.xml (deflated 42%) 2022-08-17T13:49:57.0485987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134135.xml (deflated 42%) 2022-08-17T13:49:57.0486335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134138.xml (deflated 41%) 2022-08-17T13:49:57.0486676Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134142.xml (deflated 42%) 2022-08-17T13:49:57.0487022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134146.xml (deflated 42%) 2022-08-17T13:49:57.0487363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134149.xml (deflated 41%) 2022-08-17T13:49:57.0487706Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134153.xml (deflated 42%) 2022-08-17T13:49:57.0488049Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134157.xml (deflated 42%) 2022-08-17T13:49:57.0488392Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134200.xml (deflated 42%) 2022-08-17T13:49:57.0488733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134204.xml (deflated 41%) 2022-08-17T13:49:57.0489076Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134206.xml (deflated 42%) 2022-08-17T13:49:57.0489406Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134209.xml (deflated 41%) 2022-08-17T13:49:57.0489746Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134213.xml (deflated 41%) 2022-08-17T13:49:57.0490083Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134217.xml (deflated 41%) 2022-08-17T13:49:57.0490426Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134220.xml (deflated 41%) 2022-08-17T13:49:57.0490769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134222.xml (deflated 41%) 2022-08-17T13:49:57.0491114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134223.xml (deflated 42%) 2022-08-17T13:49:57.0491451Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134225.xml (deflated 41%) 2022-08-17T13:49:57.0491791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134226.xml (deflated 42%) 2022-08-17T13:49:57.0492132Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134228.xml (deflated 42%) 2022-08-17T13:49:57.0492472Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134232.xml (deflated 41%) 2022-08-17T13:49:57.0492812Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134235.xml (deflated 42%) 2022-08-17T13:49:57.0493189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134237.xml (deflated 42%) 2022-08-17T13:49:57.0493531Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134238.xml (deflated 42%) 2022-08-17T13:49:57.0493869Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134242.xml (deflated 42%) 2022-08-17T13:49:57.0494209Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134246.xml (deflated 41%) 2022-08-17T13:49:57.0494569Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134247.xml (deflated 42%) 2022-08-17T13:49:57.0494911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134251.xml (deflated 41%) 2022-08-17T13:49:57.0495254Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134254.xml (deflated 42%) 2022-08-17T13:49:57.0495597Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134258.xml (deflated 41%) 2022-08-17T13:49:57.0495936Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134302.xml (deflated 41%) 2022-08-17T13:49:57.0496312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134305.xml (deflated 42%) 2022-08-17T13:49:57.0496643Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134309.xml (deflated 42%) 2022-08-17T13:49:57.0497013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134311.xml (deflated 41%) 2022-08-17T13:49:57.0497490Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134313.xml (deflated 40%) 2022-08-17T13:49:57.0497860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134316.xml (deflated 41%) 2022-08-17T13:49:57.0498225Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134320.xml (deflated 42%) 2022-08-17T13:49:57.0498595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134324.xml (deflated 41%) 2022-08-17T13:49:57.0498973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134325.xml (deflated 40%) 2022-08-17T13:49:57.0499340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134329.xml (deflated 41%) 2022-08-17T13:49:57.0499672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134333.xml (deflated 41%) 2022-08-17T13:49:57.0500045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134337.xml (deflated 42%) 2022-08-17T13:49:57.0500408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134338.xml (deflated 41%) 2022-08-17T13:49:57.0500801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134342.xml (deflated 41%) 2022-08-17T13:49:57.0501175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134346.xml (deflated 40%) 2022-08-17T13:49:57.0501542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134350.xml (deflated 42%) 2022-08-17T13:49:57.0501934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134352.xml (deflated 41%) 2022-08-17T13:49:57.0502333Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134355.xml (deflated 40%) 2022-08-17T13:49:57.0502765Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134359.xml (deflated 41%) 2022-08-17T13:49:57.0503139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134403.xml (deflated 40%) 2022-08-17T13:49:57.0503503Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134407.xml (deflated 40%) 2022-08-17T13:49:57.0503864Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134411.xml (deflated 41%) 2022-08-17T13:49:57.0504255Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134412.xml (deflated 41%) 2022-08-17T13:49:57.0504633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134416.xml (deflated 41%) 2022-08-17T13:49:57.0505001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134420.xml (deflated 41%) 2022-08-17T13:49:57.0505366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134424.xml (deflated 41%) 2022-08-17T13:49:57.0505733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134428.xml (deflated 41%) 2022-08-17T13:49:57.0506102Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134429.xml (deflated 41%) 2022-08-17T13:49:57.0506467Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134431.xml (deflated 41%) 2022-08-17T13:49:57.0506838Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134436.xml (deflated 41%) 2022-08-17T13:49:57.0507230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134442.xml (deflated 41%) 2022-08-17T13:49:57.0507564Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134446.xml (deflated 40%) 2022-08-17T13:49:57.0507950Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134450.xml (deflated 40%) 2022-08-17T13:49:57.0508320Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134454.xml (deflated 42%) 2022-08-17T13:49:57.0508683Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134455.xml (deflated 42%) 2022-08-17T13:49:57.0509048Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134457.xml (deflated 42%) 2022-08-17T13:49:57.0509415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134458.xml (deflated 42%) 2022-08-17T13:49:57.0509781Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134500.xml (deflated 43%) 2022-08-17T13:49:57.0510151Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134501.xml (deflated 42%) 2022-08-17T13:49:57.0510540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134503.xml (deflated 42%) 2022-08-17T13:49:57.0510904Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134504.xml (deflated 42%) 2022-08-17T13:49:57.0511272Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134506.xml (deflated 41%) 2022-08-17T13:49:57.0511634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134509.xml (deflated 42%) 2022-08-17T13:49:57.0511996Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134513.xml (deflated 41%) 2022-08-17T13:49:57.0512363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134515.xml (deflated 43%) 2022-08-17T13:49:57.0512756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134516.xml (deflated 42%) 2022-08-17T13:49:57.0513124Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134518.xml (deflated 42%) 2022-08-17T13:49:57.0513487Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134522.xml (deflated 42%) 2022-08-17T13:49:57.0513896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134526.xml (deflated 41%) 2022-08-17T13:49:57.0514262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134529.xml (deflated 41%) 2022-08-17T13:49:57.0514611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134533.xml (deflated 41%) 2022-08-17T13:49:57.0534964Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134537.xml (deflated 41%) 2022-08-17T13:49:57.0535337Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134540.xml (deflated 40%) 2022-08-17T13:49:57.0535701Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134544.xml (deflated 41%) 2022-08-17T13:49:57.0536056Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134548.xml (deflated 40%) 2022-08-17T13:49:57.0536408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134552.xml (deflated 40%) 2022-08-17T13:49:57.0536759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134556.xml (deflated 40%) 2022-08-17T13:49:57.0537108Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134559.xml (deflated 40%) 2022-08-17T13:49:57.0537454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134603.xml (deflated 40%) 2022-08-17T13:49:57.0537795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134607.xml (deflated 41%) 2022-08-17T13:49:57.0538145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134611.xml (deflated 40%) 2022-08-17T13:49:57.0538483Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134615.xml (deflated 41%) 2022-08-17T13:49:57.0538831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134619.xml (deflated 41%) 2022-08-17T13:49:57.0539176Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134622.xml (deflated 41%) 2022-08-17T13:49:57.0539522Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134626.xml (deflated 43%) 2022-08-17T13:49:57.0539863Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134628.xml (deflated 41%) 2022-08-17T13:49:57.0540336Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134632.xml (deflated 41%) 2022-08-17T13:49:57.0540680Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134636.xml (deflated 42%) 2022-08-17T13:49:57.0541025Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134637.xml (deflated 42%) 2022-08-17T13:49:57.0541363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134639.xml (deflated 41%) 2022-08-17T13:49:57.0541742Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134642.xml (deflated 41%) 2022-08-17T13:49:57.0542077Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134646.xml (deflated 41%) 2022-08-17T13:49:57.0542420Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134650.xml (deflated 41%) 2022-08-17T13:49:57.0542861Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134654.xml (deflated 41%) 2022-08-17T13:49:57.0543210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134655.xml (deflated 41%) 2022-08-17T13:49:57.0543548Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134657.xml (deflated 40%) 2022-08-17T13:49:57.0543893Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134701.xml (deflated 41%) 2022-08-17T13:49:57.0544235Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134705.xml (deflated 41%) 2022-08-17T13:49:57.0544576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134708.xml (deflated 41%) 2022-08-17T13:49:57.0544918Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134712.xml (deflated 41%) 2022-08-17T13:49:57.0545258Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134716.xml (deflated 41%) 2022-08-17T13:49:57.0545599Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134719.xml (deflated 41%) 2022-08-17T13:49:57.0545934Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134723.xml (deflated 40%) 2022-08-17T13:49:57.0546276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134727.xml (deflated 41%) 2022-08-17T13:49:57.0546620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134728.xml (deflated 41%) 2022-08-17T13:49:57.0546961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134730.xml (deflated 41%) 2022-08-17T13:49:57.0547297Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134732.xml (deflated 41%) 2022-08-17T13:49:57.0547638Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134735.xml (deflated 41%) 2022-08-17T13:49:57.0547982Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134739.xml (deflated 41%) 2022-08-17T13:49:57.0548317Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134743.xml (deflated 41%) 2022-08-17T13:49:57.0548659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134746.xml (deflated 41%) 2022-08-17T13:49:57.0549030Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134750.xml (deflated 42%) 2022-08-17T13:49:57.0549364Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134754.xml (deflated 42%) 2022-08-17T13:49:57.0549698Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134758.xml (deflated 40%) 2022-08-17T13:49:57.0550035Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134801.xml (deflated 41%) 2022-08-17T13:49:57.0550401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134805.xml (deflated 42%) 2022-08-17T13:49:57.0550744Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134809.xml (deflated 41%) 2022-08-17T13:49:57.0551088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220817134810.xml (deflated 41%) 2022-08-17T13:49:57.0551298Z adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%) 2022-08-17T13:49:57.0582798Z ##[group]Run # Remove any previous test reports if they exist 2022-08-17T13:49:57.0582962Z # Remove any previous test reports if they exist 2022-08-17T13:49:57.0583053Z rm -f usage-log-*.zip 2022-08-17T13:49:57.0583214Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2022-08-17T13:49:57.0583329Z # so check to see if the file exists first 2022-08-17T13:49:57.0583472Z if [ -f 'usage_log.txt' ]; then 2022-08-17T13:49:57.0583653Z  zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt' 2022-08-17T13:49:57.0583734Z fi 2022-08-17T13:49:57.0594766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:49:57.0594836Z env: 2022-08-17T13:49:57.0594929Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:57.0595076Z FILE_SUFFIX: test-distributed-1-1-linux.2xlarge_7878166005 2022-08-17T13:49:57.0595151Z ##[endgroup] 2022-08-17T13:49:57.1290634Z adding: usage_log.txt (deflated 94%) 2022-08-17T13:49:57.1329196Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-08-17T13:49:57.1329272Z with: 2022-08-17T13:49:57.1329394Z s3-prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:57.1329479Z retention-days: 14 2022-08-17T13:49:57.1329575Z if-no-files-found: warn 2022-08-17T13:49:57.1329663Z path: test-jsons-*.zip 2022-08-17T13:49:57.1329735Z name: artifact 2022-08-17T13:49:57.1329821Z s3-bucket: gha-artifacts 2022-08-17T13:49:57.1329897Z region: us-east-1 2022-08-17T13:49:57.1329960Z env: 2022-08-17T13:49:57.1330049Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:57.1330122Z ##[endgroup] 2022-08-17T13:49:57.4834303Z NOTE: s3-prefix specified, ignoring name parameter 2022-08-17T13:49:57.4834813Z With the provided path, there will be 1 file uploaded 2022-08-17T13:49:57.4835090Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:57.4873754Z Starting upload of test-jsons-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:57.6097721Z Finished upload of test-jsons-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:57.6220800Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-08-17T13:49:57.6221023Z with: 2022-08-17T13:49:57.6221232Z s3-prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:57.6221448Z retention-days: 14 2022-08-17T13:49:57.6221652Z if-no-files-found: error 2022-08-17T13:49:57.6221876Z path: test-reports-*.zip 2022-08-17T13:49:57.6222057Z name: artifact 2022-08-17T13:49:57.6222245Z s3-bucket: gha-artifacts 2022-08-17T13:49:57.6222442Z region: us-east-1 2022-08-17T13:49:57.6222707Z env: 2022-08-17T13:49:57.6222874Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:57.6223155Z ##[endgroup] 2022-08-17T13:49:57.9436366Z NOTE: s3-prefix specified, ignoring name parameter 2022-08-17T13:49:57.9436827Z With the provided path, there will be 1 file uploaded 2022-08-17T13:49:57.9437275Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:57.9445082Z Starting upload of test-reports-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:58.1001767Z Finished upload of test-reports-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:58.1124601Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-08-17T13:49:58.1124826Z with: 2022-08-17T13:49:58.1125039Z s3-prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:58.1125346Z retention-days: 14 2022-08-17T13:49:58.1125557Z if-no-files-found: ignore 2022-08-17T13:49:58.1125767Z path: usage-log-*.zip 2022-08-17T13:49:58.1126190Z name: artifact 2022-08-17T13:49:58.1126383Z s3-bucket: gha-artifacts 2022-08-17T13:49:58.1126582Z region: us-east-1 2022-08-17T13:49:58.1126760Z env: 2022-08-17T13:49:58.1126927Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:58.1127118Z ##[endgroup] 2022-08-17T13:49:58.4349823Z NOTE: s3-prefix specified, ignoring name parameter 2022-08-17T13:49:58.4350399Z With the provided path, there will be 1 file uploaded 2022-08-17T13:49:58.4350700Z Uploading to s3 prefix: pytorch/pytorch/2875102080/1/artifact 2022-08-17T13:49:58.4357476Z Starting upload of usage-log-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:58.6001091Z Finished upload of usage-log-test-distributed-1-1-linux.2xlarge_7878166005.zip 2022-08-17T13:49:58.6126820Z ##[group]Run set -x 2022-08-17T13:49:58.6127025Z set -x 2022-08-17T13:49:58.6127258Z python3 -m pip install -r requirements.txt 2022-08-17T13:49:58.6127520Z python3 -m pip install boto3==1.19.12 2022-08-17T13:49:58.6128170Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-08-17T13:49:58.6139417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:49:58.6139658Z env: 2022-08-17T13:49:58.6139842Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:49:58.6140043Z AWS_DEFAULT_REGION: us-east-1 2022-08-17T13:49:58.6140249Z BRANCH: pull/82657 2022-08-17T13:49:58.6140448Z TEST_CONFIG: distributed 2022-08-17T13:49:58.6140629Z SHARD_NUMBER: 1 2022-08-17T13:49:58.6140851Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-08-17T13:49:58.6141076Z PR_NUMBER: 82657 2022-08-17T13:49:58.6141258Z PYTORCH_RETRY_TEST_CASES: 1 2022-08-17T13:49:58.6141474Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-08-17T13:49:58.6141748Z SHA1: ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T13:49:58.6141947Z TAG: 2022-08-17T13:49:58.6142119Z WORKFLOW_ID: 2875102080 2022-08-17T13:49:58.6142455Z GITHUB_TOKEN: *** 2022-08-17T13:49:58.6142753Z GHA_WORKFLOW_JOB_ID: 7878166005 2022-08-17T13:49:58.6142954Z ##[endgroup] 2022-08-17T13:49:58.6168094Z + python3 -m pip install -r requirements.txt 2022-08-17T13:49:58.8327932Z Defaulting to user installation because normal site-packages is not writeable 2022-08-17T13:49:58.8973710Z Collecting astunparse 2022-08-17T13:49:58.9091735Z Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) 2022-08-17T13:49:58.9315685Z Collecting expecttest 2022-08-17T13:49:58.9346449Z Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB) 2022-08-17T13:49:58.9623916Z Collecting future 2022-08-17T13:49:58.9654487Z Downloading future-0.18.2.tar.gz (829 kB) 2022-08-17T13:49:59.8496082Z Collecting numpy 2022-08-17T13:49:59.8538214Z Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) 2022-08-17T13:50:00.0917729Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (5.9.1) 2022-08-17T13:50:00.1760596Z Collecting pyyaml 2022-08-17T13:50:00.1799540Z Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB) 2022-08-17T13:50:00.1957131Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0) 2022-08-17T13:50:00.2077110Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3) 2022-08-17T13:50:00.2513286Z Collecting six 2022-08-17T13:50:00.2543505Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2022-08-17T13:50:00.2788523Z Collecting types-dataclasses 2022-08-17T13:50:00.2818113Z Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB) 2022-08-17T13:50:00.3123590Z Collecting typing_extensions 2022-08-17T13:50:00.3154962Z Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB) 2022-08-17T13:50:00.3537760Z Collecting sympy 2022-08-17T13:50:00.3582495Z Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB) 2022-08-17T13:50:00.5165804Z Collecting wheel<1.0,>=0.23.0 2022-08-17T13:50:00.5199027Z Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) 2022-08-17T13:50:00.5289718Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12) 2022-08-17T13:50:00.5308755Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2022.6.15) 2022-08-17T13:50:00.5316916Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.11) 2022-08-17T13:50:00.5470978Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3) 2022-08-17T13:50:00.5667005Z Collecting mpmath>=0.19 2022-08-17T13:50:00.5725770Z Downloading mpmath-1.2.1-py3-none-any.whl (532 kB) 2022-08-17T13:50:00.5915523Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed. 2022-08-17T13:50:00.6278791Z Installing collected packages: wheel, six, astunparse, expecttest, future, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy 2022-08-17T13:50:00.6497302Z WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-08-17T13:50:00.6498052Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-08-17T13:50:00.6835187Z Running setup.py install for future: started 2022-08-17T13:50:01.1752116Z Running setup.py install for future: finished with status 'done' 2022-08-17T13:50:02.8221472Z WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-08-17T13:50:02.8222028Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-08-17T13:50:10.3176873Z WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-08-17T13:50:10.3177426Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-08-17T13:50:10.3391775Z Successfully installed astunparse-1.6.3 expecttest-0.1.3 future-0.18.2 mpmath-1.2.1 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.3.0 wheel-0.37.1 2022-08-17T13:50:10.4038543Z + python3 -m pip install boto3==1.19.12 2022-08-17T13:50:10.6206850Z Defaulting to user installation because normal site-packages is not writeable 2022-08-17T13:50:11.2452315Z Collecting boto3==1.19.12 2022-08-17T13:50:11.2626028Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2022-08-17T13:50:11.3009662Z Collecting jmespath<1.0.0,>=0.7.1 2022-08-17T13:50:11.3040585Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2022-08-17T13:50:12.0449099Z Collecting botocore<1.23.0,>=1.22.12 2022-08-17T13:50:12.0495853Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2022-08-17T13:50:12.2024751Z Collecting s3transfer<0.6.0,>=0.5.0 2022-08-17T13:50:12.2056700Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2022-08-17T13:50:12.2509565Z Collecting python-dateutil<3.0.0,>=2.1 2022-08-17T13:50:12.2554699Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2022-08-17T13:50:12.2677415Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.11) 2022-08-17T13:50:12.2831930Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2022-08-17T13:50:12.3473673Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3 2022-08-17T13:50:12.9916468Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 2022-08-17T13:50:13.0427878Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-08-17T13:50:21.2674520Z [scribe] Scribe access token not provided, sending report via boto3... 2022-08-17T13:50:21.2674746Z 2022-08-17T13:50:21.2675022Z ----- Historic stats comparison result ------ 2022-08-17T13:50:21.2675186Z 2022-08-17T13:50:21.2677500Z job: linux-focal-py3.7-gcc7 2022-08-17T13:50:21.2677800Z commit: ce6a3c605df99d1df57c0dda75c06d748e54ed2a 2022-08-17T13:50:21.2677965Z 2022-08-17T13:50:21.2678139Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-08-17T13:50:21.2678327Z 2022-08-17T13:50:21.2678631Z : (master) 2022-08-17T13:50:21.2678789Z | 2022-08-17T13:50:21.2678993Z | * ce6a3c605d (HEAD) total time 3068.04s 2022-08-17T13:50:21.2679194Z | | 2022-08-17T13:50:21.2679345Z | : (3 commits) 2022-08-17T13:50:21.2679512Z |/| 2022-08-17T13:50:21.2679671Z | : (6 commits) 2022-08-17T13:50:21.2679823Z | 2022-08-17T13:50:21.2681532Z * 343b5f8651 (base) 5 reports, total time 2488.33s ± 2005.09s 2022-08-17T13:50:21.2681910Z * 1a09b05c94 5 reports, total time 2470.84s ± 1972.56s 2022-08-17T13:50:21.2682242Z * df62ea76d1 5 reports, total time 2484.45s ± 1976.41s 2022-08-17T13:50:21.2682560Z * aac622ad55 5 reports, total time 2485.07s ± 2017.11s 2022-08-17T13:50:21.2682890Z * 31d4b6f52a 5 reports, total time 2575.69s ± 2119.02s 2022-08-17T13:50:21.2683220Z * 0e2efaf9cc 5 reports, total time 2933.75s ± 2779.34s 2022-08-17T13:50:21.2683562Z * 1ee9eb52b6 5 reports, total time 3224.96s ± 2891.08s 2022-08-17T13:50:21.2683794Z * f4b7c10e14 0 reports 2022-08-17T13:50:21.2684093Z * 785f7f6298 5 reports, total time 2501.75s ± 2024.28s 2022-08-17T13:50:21.2684399Z * 3586af8adc 5 reports, total time 2510.83s ± 2030.15s 2022-08-17T13:50:21.2684605Z | 2022-08-17T13:50:21.2684757Z : 2022-08-17T13:50:21.2684852Z 2022-08-17T13:50:21.2684964Z Removed (across 720 suites) 0 tests, totaling 0.00s 2022-08-17T13:50:21.2685239Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-08-17T13:50:21.2685505Z Added (across 158 suites) 2019 tests, totaling +3478.89s 2022-08-17T13:50:21.3308754Z Prepare all required actions 2022-08-17T13:50:21.3329054Z ##[group]Run ./.github/actions/teardown-linux 2022-08-17T13:50:21.3329273Z with: 2022-08-17T13:50:21.3329421Z env: 2022-08-17T13:50:21.3329602Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:50:21.3329793Z ##[endgroup] 2022-08-17T13:50:21.3345185Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh 2022-08-17T13:50:21.3345456Z .github/scripts/wait_for_ssh_to_drain.sh 2022-08-17T13:50:21.3356863Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:50:21.3357075Z env: 2022-08-17T13:50:21.3357259Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:50:21.3357455Z ##[endgroup] 2022-08-17T13:50:21.3408142Z Holding runner for 2 hours until all ssh sessions have logged out 2022-08-17T13:50:21.3470492Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-08-17T13:50:21.3470911Z # ignore expansion of "docker ps -q" since it could be empty 2022-08-17T13:50:21.3471207Z # shellcheck disable=SC2046 2022-08-17T13:50:21.3471428Z docker stop $(docker ps -q) || true 2022-08-17T13:50:21.3471666Z # Prune all of the docker images 2022-08-17T13:50:21.3471891Z docker system prune -af 2022-08-17T13:50:21.3482569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-08-17T13:50:21.3482797Z env: 2022-08-17T13:50:21.3482981Z GIT_DEFAULT_BRANCH: master 2022-08-17T13:50:21.3483163Z ##[endgroup] 2022-08-17T13:50:22.3764954Z b92e618f4d32 2022-08-17T13:50:22.8971951Z Deleted Containers: 2022-08-17T13:50:22.8972266Z b92e618f4d3253d0f3d67ca620cf26031d749b51c0f266aa01cec17e92ef61f4 2022-08-17T13:50:22.8972461Z 2022-08-17T13:50:25.3931493Z Deleted Images: 2022-08-17T13:50:25.3932793Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:a347f7e7645f04fc68e4f87c73cf0385233153b8 2022-08-17T13:50:25.3934076Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:594372be36f848ede98fd4597227797d5fa59cc4886edb7b024f352b98e83e00 2022-08-17T13:50:25.3934945Z deleted: sha256:fb4c52c9a9c8679dec38b68a7227616f7f41963ed9ff1a94b43b6c75db8d7fa9 2022-08-17T13:50:25.3935936Z deleted: sha256:2d0cb2547b4b9d8c0f0fc36a7ada2414ec9c7ebe26ac314329ddb7e1de17c69e 2022-08-17T13:50:25.3936522Z deleted: sha256:cc1340d7adf94b69ab2c9bff768e58cdd4d90642578cc8d4ea7f66da4cf23112 2022-08-17T13:50:25.3937003Z deleted: sha256:2b53265715948c54131d502fa8411815ec14b774a174417b5d22ef95971e7607 2022-08-17T13:50:25.3937498Z deleted: sha256:af6a91d2dd2cdd053bec86007fbc8bd5aef700d1f3ef930e9e22754cd97df924 2022-08-17T13:50:25.3937996Z deleted: sha256:2038eb381b9d1d80921507af4ea6c25c5582317b818aed4519202ff9584064ba 2022-08-17T13:50:25.3938495Z deleted: sha256:1dcdae05f2b226ec30f553376a83bfce0dbdf1ee39e078f881ed36d9fe65fb05 2022-08-17T13:50:25.3955339Z deleted: sha256:8ae9c853740306f9590b2659a9fc5eded4c272aafa6ca44c68f1ac1399a73e9d 2022-08-17T13:50:25.3955905Z deleted: sha256:8f612883fb0c9ea2471913600953f02d71cb0cb9193bbba140181474020c289b 2022-08-17T13:50:25.3956520Z deleted: sha256:8221e32a6bff1df86fb9a9a3a7a7470b6e6c856e2e15d557ee7dd80cc8ac24d8 2022-08-17T13:50:25.3957177Z deleted: sha256:4d84d7e8f5272ca06346a623a6e2befb2f8edc0d33ca6daf13ba33910510bf85 2022-08-17T13:50:25.3957821Z deleted: sha256:4b0c58d507f1ecd6bd478a50970d9c747ef4e2ce6fc1b7d2b80baf5f600cfe62 2022-08-17T13:50:25.3958306Z deleted: sha256:141213ff2809e06b118ff9eddbe2ec020e9c6d12b0e43da0c1e7b88c58d0f24a 2022-08-17T13:50:25.3958669Z deleted: sha256:e5fafbe3aac06b308b1cdb6938a0233c8abaa3229e3eef32dbca7cd6ef7f2bf7 2022-08-17T13:50:25.3959109Z deleted: sha256:01cc8d1326c081adec11dd67dc86f0d4d5f776d8cbf24ed9d69532d194452c29 2022-08-17T13:50:25.3959458Z deleted: sha256:a2f16c946b5d1f79c8b1a56dc41ea1dd5ad4c5e878d019093e299fa1ad9f947c 2022-08-17T13:50:25.3959869Z deleted: sha256:9d7165cf550ddcb549ad533e22bd404661e2f59765680a98d05370482edf3ae8 2022-08-17T13:50:25.3960204Z deleted: sha256:dd96707e0bc09ea43a4f200ef9dad8327a52200126e830dc9f678f6499835b80 2022-08-17T13:50:25.3960778Z deleted: sha256:4a8bc9baed7e3c646f1791e45fcc9158d4b49d5126a38c089fd743a9d996dd55 2022-08-17T13:50:25.3961107Z deleted: sha256:988088eda79f6d72724b7321dd7dd84674b286c73064b3f330ee76599d75a686 2022-08-17T13:50:25.3961516Z deleted: sha256:2374db1c0d6af1d3fda08553755550caa403d05bd962525deadb1e79ef6dc3a9 2022-08-17T13:50:25.3961870Z deleted: sha256:38c4ccb6efb209d5d1d7fe4d692c33e6778e04d0b049a505b970ac80647c8021 2022-08-17T13:50:25.3965713Z deleted: sha256:2bcc0c4f43849c357f23b0f8e492ea2d6b1a48f4ff4da7bb1ff23668ee07f794 2022-08-17T13:50:25.3966427Z deleted: sha256:ef60874b6600721074a89182041a8f8e1b8ff4c9572a7de5d1b69e89ce8780be 2022-08-17T13:50:25.3966983Z deleted: sha256:8f58e1722c3247e2107f12f6a7aa773d569c9cd7148281b5fa119270ee0e301d 2022-08-17T13:50:25.3967582Z deleted: sha256:f50690cb9e6bdf8228aca3df995490a49bad2f2346fd2b4cb16aaf2bb3935293 2022-08-17T13:50:25.3968286Z deleted: sha256:1d0892db23be9ad9bfc1107a6c67f40aa11623428758509038c105a5937ff9f3 2022-08-17T13:50:25.3968891Z deleted: sha256:043f10238c4fdad7165d655316155eeeb475e8f365fa17126a694c4521d4ae79 2022-08-17T13:50:25.3969739Z deleted: sha256:553b5f6dfe83a94efad47e40546556b505415957bcc1663ff9c9569474e7e5ad 2022-08-17T13:50:25.4006442Z deleted: sha256:37753eb9cdc0f259b223d407ea0c9a9f189d0e6f8fc75bba7bfe7b8bb6107360 2022-08-17T13:50:25.4006808Z deleted: sha256:474d4748a6db06332a7f40faa666d4b24cf2407c0122e2aa2087390f65f4dbd1 2022-08-17T13:50:25.4007143Z deleted: sha256:1ede402f033e02651f0c166142559bc4611e9c1064af65d7f20e7c83f50c9a64 2022-08-17T13:50:25.4007490Z deleted: sha256:801cc279fd42150f0d92def07783b1acd9c14e7f275afbd6d17bdd3f13253186 2022-08-17T13:50:25.4007837Z deleted: sha256:eee9b1cc2de733433acd46d45ae3334201cd0565bd7c708b9cdfba46964f14f9 2022-08-17T13:50:25.4008183Z deleted: sha256:53fb226ef291c5fd0368a0463c06418b9b6bdc9f9216e621191b7658c857b345 2022-08-17T13:50:25.4008517Z deleted: sha256:4b78144a1e3f787fc836428a41b3f814b6a778fdad392680d573514fb5b9f2c7 2022-08-17T13:50:25.4008879Z deleted: sha256:f9edf9a450f9ddcf7efc2b42fa77dd9a37bc0207ffedbedce3536a4954990321 2022-08-17T13:50:25.4009238Z deleted: sha256:ecd1bd81a07b193ea24963c873e861324def6234a394b0cb1028d21f4f3641e7 2022-08-17T13:50:25.4009581Z deleted: sha256:01898bcfbc799c6b94e0def2782a011c2e100aa9e2a27f6eb7429684f3748770 2022-08-17T13:50:25.4009919Z deleted: sha256:682b0b412fc7c00587b773aa6a33bf184c28814b7036dc88c022c9d45c0a437c 2022-08-17T13:50:25.4010249Z deleted: sha256:57fce6ee1a3613c7040f3ef361aa48be78abd9a4105fe439c95e386bb12d13a0 2022-08-17T13:50:25.4010611Z deleted: sha256:51bf4e16b50a0652e9491556b6cea015da597559b58a507b67cebfe8a688ae3d 2022-08-17T13:50:25.4010947Z deleted: sha256:c24d44cff78a90fb4905a141bb9e6a7fb59a6dd63a974436952f9ec590078426 2022-08-17T13:50:25.4011294Z deleted: sha256:39ab34fc8a2a9f2aa438abef98d79822a33c30dbf03980a45c62f027253f3c7e 2022-08-17T13:50:25.4011634Z deleted: sha256:90dbe860572cf2e3d5e0b324bd84e2e9d1b6293ff3a180bec3b9419469e16996 2022-08-17T13:50:25.4011991Z deleted: sha256:39ee4dbd2638c415fb28b35c14fe55dacb2aab2c7ca47bef8407c839b5c69bdf 2022-08-17T13:50:25.4012354Z deleted: sha256:fadd72835ebf8d726a99a39b82c26aa1a2f443bc6892b300ac9e42ca9bc01fdf 2022-08-17T13:50:25.4012717Z deleted: sha256:245afb53fdc4b0ff4caa5a68d68722a4c341eed2284179368feef358e4018f2d 2022-08-17T13:50:25.4013046Z deleted: sha256:215f0022beb0eec34838c527256a6a32bde4281e1c3e1506c97881354f539786 2022-08-17T13:50:25.4013369Z deleted: sha256:775f18a80d2412c731729719e69143252dd4b2f4f2f556a5f1ded320ca220de6 2022-08-17T13:50:25.4013679Z deleted: sha256:b140655968278b483da972e18ea3578ea839d9f05e78843809c8b546ed899834 2022-08-17T13:50:25.4013990Z deleted: sha256:107411100b28d8ebcd204ea5baca21a2093e1e48e8aac5576898abc4295704ec 2022-08-17T13:50:25.4014321Z deleted: sha256:8fde7a40967705a136524cea4a68d44ae0a155f069680fab698e5555d9b96d34 2022-08-17T13:50:25.4014666Z deleted: sha256:d6e6a8cab95ba133a8182ec71cdb7cea77e4b02173aa90ca3af0300f55a84fd0 2022-08-17T13:50:25.4015023Z deleted: sha256:08bf5babfd0b448a8d39227aff88e6c3b10f48d9651561e742e668912b3dbc5c 2022-08-17T13:50:25.4015496Z deleted: sha256:734f2fd68c5f68d7f9691bc1502d405aca5054c090668a7f86201ab189801118 2022-08-17T13:50:25.4015839Z deleted: sha256:4f4b7ba1e23c2573eda42a3170ec1f097cd4fb77b4ff142ce9e7a4d08feb08cb 2022-08-17T13:50:25.4016258Z deleted: sha256:21c2921e9c1426096c1faf5e1bee42e4c78b43550961584e000bf8afa3c91440 2022-08-17T13:50:25.4016592Z deleted: sha256:0c0d4ea4a7f43436cc2b7785ac11cf898afcaf3c0ee1b9e52e103bffa9b32ebf 2022-08-17T13:50:25.4016949Z deleted: sha256:c3f11d77a5de76ec836c52333d45ac3742c7b27d3d83feba6ec978e223715c67 2022-08-17T13:50:25.4017142Z 2022-08-17T13:50:25.4117297Z Total reclaimed space: 7.17GB 2022-08-17T13:50:25.4171345Z Post job cleanup. 2022-08-17T13:50:25.4201337Z Post job cleanup. 2022-08-17T13:50:25.7599018Z [command]/usr/bin/git version 2022-08-17T13:50:25.7650494Z git version 2.37.1 2022-08-17T13:50:25.7719560Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/475f8e37-232f-4e8a-bdc7-e8fdbd12b42b' before making global git config changes 2022-08-17T13:50:25.7720253Z Adding repository directory to the temporary git global config as a safe directory 2022-08-17T13:50:25.7726259Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-08-17T13:50:25.7773037Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-08-17T13:50:25.7805655Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-08-17T13:50:25.8132035Z Entering 'android/libs/fbjni' 2022-08-17T13:50:25.8166722Z Entering 'third_party/FP16' 2022-08-17T13:50:25.8202540Z Entering 'third_party/FXdiv' 2022-08-17T13:50:25.8236977Z Entering 'third_party/NNPACK' 2022-08-17T13:50:25.8272722Z Entering 'third_party/QNNPACK' 2022-08-17T13:50:25.8307060Z Entering 'third_party/XNNPACK' 2022-08-17T13:50:25.8351000Z Entering 'third_party/benchmark' 2022-08-17T13:50:25.8385678Z Entering 'third_party/cpuinfo' 2022-08-17T13:50:25.8420184Z Entering 'third_party/cub' 2022-08-17T13:50:25.8454434Z Entering 'third_party/cudnn_frontend' 2022-08-17T13:50:25.8493835Z Entering 'third_party/eigen' 2022-08-17T13:50:25.8530666Z Entering 'third_party/fbgemm' 2022-08-17T13:50:25.8565285Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T13:50:25.8599324Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T13:50:25.8634403Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T13:50:25.8670262Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T13:50:25.8705062Z Entering 'third_party/flatbuffers' 2022-08-17T13:50:25.8743053Z Entering 'third_party/fmt' 2022-08-17T13:50:25.8777738Z Entering 'third_party/foxi' 2022-08-17T13:50:25.8812040Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T13:50:25.8845732Z Entering 'third_party/gloo' 2022-08-17T13:50:25.8879860Z Entering 'third_party/googletest' 2022-08-17T13:50:25.8914454Z Entering 'third_party/ideep' 2022-08-17T13:50:25.8948232Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T13:50:25.8983979Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T13:50:25.9023559Z Entering 'third_party/ios-cmake' 2022-08-17T13:50:25.9057074Z Entering 'third_party/ittapi' 2022-08-17T13:50:25.9091240Z Entering 'third_party/kineto' 2022-08-17T13:50:25.9124971Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T13:50:25.9158473Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T13:50:25.9192403Z Entering 'third_party/nccl/nccl' 2022-08-17T13:50:25.9226443Z Entering 'third_party/neon2sse' 2022-08-17T13:50:25.9259882Z Entering 'third_party/nlohmann' 2022-08-17T13:50:25.9295376Z Entering 'third_party/onnx' 2022-08-17T13:50:25.9341433Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T13:50:25.9375835Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T13:50:25.9412572Z Entering 'third_party/onnx-tensorrt' 2022-08-17T13:50:25.9446310Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T13:50:25.9484816Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T13:50:25.9518614Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T13:50:25.9552954Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T13:50:25.9592377Z Entering 'third_party/pocketfft' 2022-08-17T13:50:25.9627097Z Entering 'third_party/protobuf' 2022-08-17T13:50:25.9664622Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T13:50:25.9698466Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T13:50:25.9734297Z Entering 'third_party/psimd' 2022-08-17T13:50:25.9768693Z Entering 'third_party/pthreadpool' 2022-08-17T13:50:25.9803069Z Entering 'third_party/pybind11' 2022-08-17T13:50:25.9837347Z Entering 'third_party/python-enum' 2022-08-17T13:50:25.9872012Z Entering 'third_party/python-peachpy' 2022-08-17T13:50:25.9906169Z Entering 'third_party/python-six' 2022-08-17T13:50:25.9940686Z Entering 'third_party/sleef' 2022-08-17T13:50:25.9976278Z Entering 'third_party/tbb' 2022-08-17T13:50:26.0012372Z Entering 'third_party/tensorpipe' 2022-08-17T13:50:26.0047447Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T13:50:26.0081421Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T13:50:26.0115940Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T13:50:26.0151038Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T13:50:26.0185367Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T13:50:26.0221684Z Entering 'third_party/zstd' 2022-08-17T13:50:26.0274605Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-08-17T13:50:26.0301495Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0309432Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-08-17T13:50:26.0343289Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-08-17T13:50:26.0601444Z Entering 'android/libs/fbjni' 2022-08-17T13:50:26.0620958Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0649844Z Entering 'third_party/FP16' 2022-08-17T13:50:26.0670434Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0697138Z Entering 'third_party/FXdiv' 2022-08-17T13:50:26.0717695Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0747433Z Entering 'third_party/NNPACK' 2022-08-17T13:50:26.0768513Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0796166Z Entering 'third_party/QNNPACK' 2022-08-17T13:50:26.0816410Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0844047Z Entering 'third_party/XNNPACK' 2022-08-17T13:50:26.0864420Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0901979Z Entering 'third_party/benchmark' 2022-08-17T13:50:26.0922498Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0949250Z Entering 'third_party/cpuinfo' 2022-08-17T13:50:26.0969492Z http.https://github.com/.extraheader 2022-08-17T13:50:26.0996268Z Entering 'third_party/cub' 2022-08-17T13:50:26.1016514Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1043494Z Entering 'third_party/cudnn_frontend' 2022-08-17T13:50:26.1064420Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1097107Z Entering 'third_party/eigen' 2022-08-17T13:50:26.1117041Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1147230Z Entering 'third_party/fbgemm' 2022-08-17T13:50:26.1168703Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1195426Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-08-17T13:50:26.1215334Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1242282Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-08-17T13:50:26.1264386Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1291803Z Entering 'third_party/fbgemm/third_party/googletest' 2022-08-17T13:50:26.1311341Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1338668Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-08-17T13:50:26.1359204Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1388245Z Entering 'third_party/flatbuffers' 2022-08-17T13:50:26.1409829Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1438493Z Entering 'third_party/fmt' 2022-08-17T13:50:26.1458936Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1486762Z Entering 'third_party/foxi' 2022-08-17T13:50:26.1507594Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1535013Z Entering 'third_party/gemmlowp/gemmlowp' 2022-08-17T13:50:26.1555181Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1582303Z Entering 'third_party/gloo' 2022-08-17T13:50:26.1602704Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1630358Z Entering 'third_party/googletest' 2022-08-17T13:50:26.1650729Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1677159Z Entering 'third_party/ideep' 2022-08-17T13:50:26.1697604Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1723928Z Entering 'third_party/ideep/mkl-dnn' 2022-08-17T13:50:26.1744321Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1772739Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-08-17T13:50:26.1792952Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1826179Z Entering 'third_party/ios-cmake' 2022-08-17T13:50:26.1847582Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1874460Z Entering 'third_party/ittapi' 2022-08-17T13:50:26.1897128Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1924172Z Entering 'third_party/kineto' 2022-08-17T13:50:26.1944772Z http.https://github.com/.extraheader 2022-08-17T13:50:26.1972112Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-08-17T13:50:26.1991898Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2018939Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-08-17T13:50:26.2038524Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2066607Z Entering 'third_party/nccl/nccl' 2022-08-17T13:50:26.2086915Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2113645Z Entering 'third_party/neon2sse' 2022-08-17T13:50:26.2134231Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2160736Z Entering 'third_party/nlohmann' 2022-08-17T13:50:26.2181319Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2209999Z Entering 'third_party/onnx' 2022-08-17T13:50:26.2229792Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2269069Z Entering 'third_party/onnx/third_party/benchmark' 2022-08-17T13:50:26.2290217Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2318113Z Entering 'third_party/onnx/third_party/pybind11' 2022-08-17T13:50:26.2338257Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2367700Z Entering 'third_party/onnx-tensorrt' 2022-08-17T13:50:26.2388502Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2416060Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-08-17T13:50:26.2435649Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2467356Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-08-17T13:50:26.2487794Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2515460Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-08-17T13:50:26.2537202Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2564165Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-08-17T13:50:26.2584937Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2617121Z Entering 'third_party/pocketfft' 2022-08-17T13:50:26.2637649Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2664984Z Entering 'third_party/protobuf' 2022-08-17T13:50:26.2685426Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2715616Z Entering 'third_party/protobuf/third_party/benchmark' 2022-08-17T13:50:26.2736012Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2762649Z Entering 'third_party/protobuf/third_party/googletest' 2022-08-17T13:50:26.2783432Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2811909Z Entering 'third_party/psimd' 2022-08-17T13:50:26.2831985Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2859179Z Entering 'third_party/pthreadpool' 2022-08-17T13:50:26.2879476Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2907623Z Entering 'third_party/pybind11' 2022-08-17T13:50:26.2927909Z http.https://github.com/.extraheader 2022-08-17T13:50:26.2954807Z Entering 'third_party/python-enum' 2022-08-17T13:50:26.2976192Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3002842Z Entering 'third_party/python-peachpy' 2022-08-17T13:50:26.3023543Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3050584Z Entering 'third_party/python-six' 2022-08-17T13:50:26.3070827Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3098177Z Entering 'third_party/sleef' 2022-08-17T13:50:26.3118716Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3146148Z Entering 'third_party/tbb' 2022-08-17T13:50:26.3166696Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3195853Z Entering 'third_party/tensorpipe' 2022-08-17T13:50:26.3216764Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3243447Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-08-17T13:50:26.3263954Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3291731Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-08-17T13:50:26.3311599Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3338415Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-08-17T13:50:26.3358170Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3384936Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-08-17T13:50:26.3404726Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3431169Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-08-17T13:50:26.3451859Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3481736Z Entering 'third_party/zstd' 2022-08-17T13:50:26.3502353Z http.https://github.com/.extraheader 2022-08-17T13:50:26.3737150Z Cleaning up orphan processes